o zif6@s8ddlZddlZddlZddlmZddlmZddlZddlm Z ddl m Z ddl mZddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddl m!Z!ddl"m#Z#ddZ$GdddZ%ddZ&dde'fddZ(GdddZ)dS) N) defaultdictwraps)box)Console)Group)Live)Markdown)Padding)Panel) BarColumn)Progress) SpinnerColumn)TimeElapsedColumn)TimeRemainingColumn)Rule)Table) SummaryWritercCsgSNrrrL/home/ubuntu/.local/lib/python3.10/site-packages/audiotools/ml/decorators.py default_listsrc@s0eZdZdZddZddZddZdd Zd S) MeanzFKeeps track of the running mean, along with the latest value. cCs |dSr)resetselfrrr__init__$s z Mean.__init__cCs|jt|jd}|SN)totalmaxcount)rmeanrrr__call__'sz Mean.__call__cCsd|_d|_dS)Nr)r!rrrrrr+s z Mean.resetcCs.t|r|jd7_|j|7_dSdSr)mathisfiniter!r)rvalrrrupdate/s z Mean.updateN)__name__ __module__ __qualname____doc__rr#rr'rrrrrs  rcfdd}|S)a;Runs a function only when the condition is met. The condition is a function that is run. Parameters ---------- condition : Callable Function to run to check whether or not to run the decorated function. Example ------- Checkpoint only runs every 100 iterations, and only if the local rank is 0. >>> i = 0 >>> rank = 0 >>> >>> @when(lambda: i % 100 == 0 and rank == 0) >>> def checkpoint(): >>> print("Saving to /runs/exp1") >>> >>> for i in range(1000): >>> checkpoint() cstfdd}|S)Ncsr |i|SdSrr)argskwargs) conditionfnrr decoratedQsz*when..decorator..decoratedrr0r1r/r0r decoratorPszwhen..decoratorr)r/r5rr3rwhen5s r6timeprefixcr,)amAdds execution time to the output dictionary of the decorated function. The function decorated by this must output a dictionary. The key added will follow the form "[prefix]/[name_of_function]" Parameters ---------- prefix : str, optional The key added will follow the form "[prefix]/[name_of_function]", by default "time". cstfdd}|S)NcsHt}|i|}t|tsJt}|||dj<|S)N/)r7 perf_counter isinstancedictr()r-r.soutpute)r0r8rrr1hs z+timer..decorator..decoratedrr2r8r4rr5gsztimer..decoratorr)r8r5rr@rtimer[s  rAc @seZdZdZ     d'dedededed ef d d Zd d ZddZ dedefddZ de j j dejvfdededede j def ddZd(dededefdd Zd!d"Zd#d$Zd%d&ZdS))Trackera  A tracker class that helps to monitor the progress of training and logging the metrics. Attributes ---------- metrics : dict A dictionary containing the metrics for each label. history : dict A dictionary containing the history of metrics for each label. writer : SummaryWriter A SummaryWriter object for logging the metrics. rank : int The rank of the current process. step : int The current step of the training. tasks : dict A dictionary containing the progress bars and tables for each label. pbar : Progress A progress bar object for displaying the progress. consoles : list A list of console objects for logging. live : Live A Live object for updating the display live. Methods ------- print(msg: str) Prints the given message to all consoles. update(label: str, fn_name: str) Updates the progress bar and table for the given label. done(label: str, title: str) Resets the progress bar and table for the given label and prints the final result. track(label: str, length: int, completed: int = 0, op: dist.ReduceOp = dist.ReduceOp.AVG, ddp_active: bool = "LOCAL_RANK" in os.environ) A decorator for tracking the progress and metrics of a function. log(label: str, value_type: str = "value", history: bool = True) A decorator for logging the metrics of a function. is_best(label: str, key: str) -> bool Checks if the latest value of the given key in the label is the best so far. state_dict() -> dict Returns a dictionary containing the state of the tracker. load_state_dict(state_dict: dict) -> Tracker Loads the state of the tracker from the given state dictionary. Nrdwriterlog_filerank console_widthstepcCsi|_i|_||_||_||_i|_ttddtt dt |_ t |dg|_ t|j ddd|_|durE|j t |t|d d dSdS) a3 Initializes the Tracker object. Parameters ---------- writer : SummaryWriter, optional A SummaryWriter object for logging the metrics, by default None. log_file : str, optional The path to the log file, by default None. rank : int, optional The rank of the current process, by default 0. console_width : int, optional The width of the console, by default 100. step : int, optional The current step of the training, by default 0. z([progress.description]{task.description}z{task.completed}/{task.total}r9)widthr )consolerefresh_per_secondNa)rIfile)metricshistoryrDrFrHtasksr rr rrpbarrconsolesrliveappendopen)rrDrErFrGrHrrrrs( zTracker.__init__cCs(|jdkr|jD] }||qdSdS)z Prints the given message to all consoles. Parameters ---------- msg : str The message to be printed. rN)rFrSlog)rmsgcrrrprints  z Tracker.printc Cs0|jdkr|j|j|dt|dtjd}|jddd|jdd d|jd d d|j|d }|D] }|j|d|}|j|d |}| ||d |d q8||j|d <dd|j D}t g||jR} |j t tddtd|dddtddtj| dddddSdS)a Updates the progress bar and table for the given label. Parameters ---------- label : str The label of the progress bar and table to be updated. fn_name : str The name of the function associated with the label. rrRT)titleexpandrkeycyan)stylevalue bright_bluer" bright_greenz10.6ftablecSg|]}|dqSrcr.0trrr z"Tracker.update..)rrz[italic]z()white)rz [b]Progressblue)paddingr[ border_styleN)rFrRadvancerQrrMINIMAL add_columnrOkeysadd_rowvaluesrrTr'r rr fit) rlabelfn_namercrtkr`r"tablesgrouprrrr's2 zTracker.updaterxr[cCs|jD]}|j|dD]}|qq|jdkrG|j|j|ddd|jD}ttd|g||jR}||dSdS)a: Resets the progress bar and table for the given label and prints the final result. Parameters ---------- label : str The label of the progress bar and table to be reset. title : str The title to be displayed when printing the final result. r"rrRcSrdrerrfrrrrirjz Tracker.done..z# N) rOrvrrFrRrQrr rZ)rrxr[vr{r|rrrdones    z Tracker.done LOCAL_RANKlength completedop ddp_activecsXjjdd||dtdj<ttdddj<fdd }|S) a A decorator for tracking the progress and metrics of a function. Parameters ---------- label : str The label to be associated with the progress and metrics. length : int The total number of iterations to be completed. completed : int, optional The number of iterations already completed, by default 0. op : dist.ReduceOp, optional The reduce operation to be used, by default dist.ReduceOp.AVG. ddp_active : bool, optional Whether the DistributedDataParallel is active, by default "LOCAL_RANK" in os.environ. z[white]Iteration ())rr)rRrccSstSr)rrrrr;szTracker.track..)r`r"cs tfdd}|S)Ncs|i|}t|tsj|Sg}|D];\}}t|ttfr,t|g}t |s2qr>|j r>t j |d| ||<t|dkrV|||||<q|D]\}}||vrdq[|jd|<jd||q[j|S)N)rrr`r")r;r<r'r(itemsintfloattorchtensor is_tensoris_cudadist all_reducedetachnumelrUitemrO)r-r.r> scalar_keysrzr})rr0rxrrrrr1?s0       z3Tracker.track..decorator..decoratedrr2rrxrrr4rr5>sz Tracker.track..decorator)rRadd_taskrrQrrO)rrxrrrrr5rrrtracks   "z Tracker.trackr`T value_typerPcs<dvsJ|rjvrttj<fdd}|S)a A decorator for logging the metrics of a function. Parameters ---------- label : str The label to be associated with the logging. value_type : str, optional The type of value to be logged, by default "value". history : bool, optional Whether to save the history of the metrics, by default True. )r"r`cstfdd}|S)Ncs|i|}jdkrXj}|D]0\}}t|tr#|n|}jdur8j|d|jjvrGj| |qjvrXjd j|S)Nrr9rH) rFrOrr;rrD add_scalarrHrPrU)r-r.r>rOrzr})r0rxrrrrr1us    z1Tracker.log..decorator..decoratedrr2rxrrr4rr5tszTracker.log..decorator)rPrr)rrxrrPr5rrrrWbs  z Tracker.logcCs$|j||dt|j||kS)a Checks if the latest value of the given key in the label is the best so far. Parameters ---------- label : str The label of the metrics to be checked. key : str The key of the metric to be checked. Returns ------- bool True if the latest value is the best so far, otherwise False. )rPmin)rrxr]rrris_bests$zTracker.is_bestcCs|j|jdS)z Returns a dictionary containing the state of the tracker. Returns ------- dict A dictionary containing the history and step of the tracker. rPrHrrrrr state_dicts zTracker.state_dictcCs|d|_|d|_|S)aB Loads the state of the tracker from the given state dictionary. Parameters ---------- state_dict : dict A dictionary containing the history and step of the tracker. Returns ------- Tracker The tracker object with the loaded state. rPrHr)rrrrrload_state_dicts  zTracker.load_state_dict)NNrrCr)r`T)r(r)r*r+rstrrrrZr'r~rReduceOpAVGosenvironboolrrWrrrrrrrrBvsN. . ( G) rB)r7)*r$rr7 collectionsr functoolsrrtorch.distributed distributedrrichr rich.consolerr rich.liver rich.markdownr rich.paddingr rich.panelr rich.progressr r rrr rich.ruler rich.tablertorch.utils.tensorboardrrrr6rrArBrrrrs6                  &