o pib@sddlZddlmZmZmZmZmZmZddlZ ddl Z ddl m Z ddl m mZddlmZddlmZddlmZddlmZmZddlmZddlmZdd lmZdd lm Z dd l!m"Z"m#Z#m$Z$Gd d d Z%Gddde%Z&dS)N)CallableListOptionalTextTupleUnion) rearrange) is_oom_error) AudioFile)ModelSpecifications) Resolution)map_with_specifications)Powerset)fix_reproducibility)Segment SlidingWindowSlidingWindowFeaturec@s eZdZdS) BaseInferenceN)__name__ __module__ __qualname__rrQ/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/core/inference.pyr*src@seZdZdZ        d0dededeed eed ee j ge j fd e d e d ee j defddZd e j ddfddZde jdee j ee j ffddZde jdedeedeeeeffddZ d1dedeedeeeee j feee j fffddZ d1dedeeeefdeedeeeee j feee j fffd d!Zed"d#de jdfd$ed%ed&eeefd'ed(e d)ed*e defd+d,Ze -d2d$ed&eeefdefd.d/Z dS)3 InferenceaInference Parameters ---------- model : Model Model. Will be automatically set to eval() mode and moved to `device` when provided. window : {"sliding", "whole"}, optional Use a "sliding" window and aggregate the corresponding outputs (default) or just one (potentially long) window covering the "whole" file or chunk. duration : float, optional Chunk duration, in seconds. Defaults to duration used for training the model. Has no effect when `window` is "whole". step : float, optional Step between consecutive chunks, in seconds. Defaults to warm-up duration when greater than 0s, otherwise 10% of duration. Has no effect when `window` is "whole". pre_aggregation_hook : callable, optional When a callable is provided, it is applied to the model output, just before aggregation. Takes a (num_chunks, num_frames, dimension) numpy array as input and returns a modified (num_chunks, num_frames, other_dimension) numpy array passed to overlap-add aggregation. skip_aggregation : bool, optional Do not aggregate outputs when using "sliding" window. Defaults to False. skip_conversion: bool, optional In case a task has been trained with `powerset` mode, output is automatically converted to `multi-label`, unless `skip_conversion` is set to True. batch_size : int, optional Batch size. Larger values (should) make inference faster. Defaults to 32. device : torch.device, optional Device used for inference. Defaults to `model.device`. In case `device` and `model.device` are different, model is sent to device. slidingNF modelwindowdurationsteppre_aggregation_hookskip_aggregationskip_conversiondevice batch_sizec Cs||_|dur |jj}||_|j|j|j|jj} |dvr&td|dkr8tdd| Dr8td||_ t t | j } |pE| }| |krXtd| dd |dd ||_ ||_ t} | D]} | jrt|sttt| j| j}nt}| ||jqct| tr| d |_nt| |_||_||_t t | j|_|p|jd d krd |j n|jd }||j krtd|dd|j dd||_| |_dS)N)rwholez&`window` must be "sliding" or "whole".r&css|] }|jtjkVqdSN) resolutionr FRAME).0srrr ls  z%Inference.__init__..zUsing "whole" `window` inference with a frame-based model might lead to bad results and huge memory consumption: it is recommended to set `window` to "sliding".zModel was trained with gzs chunks, and you requested z>s chunks for inference: this might lead to suboptimal results.r皙?z*Step between consecutive chunks is set to zs, while chunks are only z^s long, leading to gaps between consecutive chunks. Either decrease step or increase duration.) rr$evaltospecifications ValueErroranywarningswarnrnextiterrr#listpowersetrlenclassespowerset_max_classesnnIdentityappend isinstancer conversion ModuleListr"r!warm_upr r%)selfrrrr r!r"r#r$r%r2training_durationrBr+crrr__init__Nsb          zInference.__init__returncCsDt|tjstdt|jd|j||j|||_|S)zSend internal model to `device`z5`device` must be an instance of `torch.device`, got ``) rAtorchr$ TypeErrortyperrr1rB)rEr$rrrr1s   z Inference.tochunksc Cst0z |||j}Wnty,}zt|r&td|jdd|d}~wwWdn1s7wYdtj dt j fdd}t |jj |||jS) ahForward pass Takes care of sending chunks to right device and outputs back to CPU Parameters ---------- chunks : (batch_size, num_channels, num_samples) torch.Tensor Batch of audio chunks. Returns ------- outputs : (tuple of) (batch_size, ...) np.ndarray Model output. z batch_size (z dzP) is probably too large. Try with a smaller value until memory error disappears.NoutputrBc[s||Sr')cpunumpy)rOrBkwargsrrr __convertsz"Inference.infer..__convert)rKinference_moderr1r$ RuntimeErrorr MemoryErrorr%Tensorr>Modulerr2rB)rErNoutputs exception_Inference__convertrrrinfers"  zInference.inferwaveform sample_ratehookc s"jjj}tj}|j\} ddttdt ffdd }t jj |jj }|krBt |d||d} | j\} }}nd} |kpO||dkro|dd| |df} | j\}} || } t| d| f} d d }t jj |}|dur|d| d dd d }td| jD](}| ||j}|}t jj |||}|dur||j| d qr܈| d}t jj |||}|dur|| | d dttjdtjfdd}t jj ||} ddtjdt dttdtffdd }t jj |||S)aSlide model on a waveform Parameters ---------- waveform: (num_channels, num_samples) torch.Tensor Waveform. sample_rate : int Sample rate. hook: Optional[Callable] When a callable is provided, it is called everytime a batch is processed with two keyword arguments: - `completed`: the number of chunks that have been processed so far - `total`: the total number of chunks Returns ------- output : (tuple of) SlidingWindowFeature Model output. Shape is (num_chunks, dimension) for chunk-level tasks, and (num_frames, dimension) for frame-level tasks. Nr2rIcs"|jtjkrtdjjdS|S)Nr.startrr )r(r CHUNKrrr )receptive_fieldr2)rErr__framess z!Inference.slide..__framesz*channel chunk frame -> chunk channel framerc[stSr')r9)rRrrr __empty_listsz%Inference.slide..__empty_list) completedtotalc[s||dSr')r@)rO batch_outputrRrrr__append_batch"s z'Inference.slide..__append_batchrOc[s t|Sr')npvstack)rOrRrrr__vstackAs z!Inference.slide..__vstackrYframescsjs|jtjks|jrjdurtdjjd}t ||Sjdur)|}j t |tdjjd|j ddd}rM|j t ddd|_|S)Nr.r`T)rDhammingmissingloose)mode)r"r(r rbpermutation_invariantr!rrr r aggregaterDcroprdata)rYrnr2 aggregatedhas_last_chunk num_samplesr^rErr __aggregateHs6        z$Inference.slide..__aggregater')rIN)raudioget_num_samplesrroundr shaperr rrr2rcrunfoldFpadrkaranger%r\rndarrayr)rEr]r^r_ window_size step_size__Inference__framesrnrN num_chunks last_chunklast_window_sizelast_pad_Inference__empty_listrY_Inference__append_batchrGbatch batch_outputs last_outputs_Inference__vstack_Inference__aggregaterrxrslides         + zInference.slidefilecCsht|j|j|\}}|jdkr|j|||dS||d}dtjdtjfdd}t |jj ||S)aRun inference on a whole file Parameters ---------- file : AudioFile Audio file. hook : callable, optional When a callable is provided, it is called everytime a batch is processed with two keyword arguments: - `completed`: the number of chunks that have been processed so far - `total`: the total number of chunks Returns ------- output : (tuple of) SlidingWindowFeature or np.ndarray Model output, as `SlidingWindowFeature` if `window` is set to "sliding" and `np.ndarray` if is set to "whole". rr_NrYrIc[|dSNrrrYrRrrr__first_samplez*Inference.__call__..__first_sample) rr$rr|rrr\rkrrr2)rErr_r]r^rY_Inference__first_samplerrr__call__ws   zInference.__call__chunkc s tjjdkrLtts'tddD}tddD}t||djj \}}j |||d}dt dt ffd d } t jj | |Sttr\jj \}}ntjfd d Dd d}|d}dtjdtjfdd} t jj | |S)aRun inference on a chunk or a list of chunks Parameters ---------- file : AudioFile Audio file. chunk : Segment or list of Segment Apply model on this chunk. When a list of chunks is provided and window is set to "sliding", this is equivalent to calling crop on the smallest chunk that contains all chunks. In case window is set to "whole", this is equivalent to concatenating each chunk into one (artifical) chunk before processing it. hook : callable, optional When a callable is provided, it is called everytime a batch is processed with two keyword arguments: - `completed`: the number of chunks that have been processed so far - `total`: the total number of chunks Returns ------- output : (tuple of) SlidingWindowFeature or np.ndarray Model output, as `SlidingWindowFeature` if `window` is set to "sliding" and `np.ndarray` if is set to "whole". Notes ----- If model needs to be warmed up, remember to extend the requested chunk with the corresponding amount of time so that it is actually warmed up when processing the chunk of interest: >>> chunk_of_interest = Segment(10, 15) >>> extended_chunk = Segment(10 - warm_up, 15 + warm_up) >>> inference.crop(file, extended_chunk).crop(chunk_of_interest, returns_data=False) rcs|]}|jVqdSr')rar*rGrrrr,z!Inference.crop..csrr')endrrrrr,r)rarrrOrIcs&|j}tj|j|jd}t|j|S)Nr`)sliding_windowrrarr rrv)rOrRrnshifted_frames)rrr__shifts   zInference.crop..__shiftcs g|] }jj|dqS)r)rr|rur)rrErr s z"Inference.crop..re)dimNrYc[rrrrrrrrrz&Inference.crop..__first_sample)rr$rrArminmaxrr|rurrrr2rKcatr\rkr) rErrr_rarr]r^rY_Inference__shiftrr)rrrErrus* +     zInference.crop)r.r.g-q=scoresrnrDepsilonrorp skip_averagecCs|jj\}}} |j} t| j|j|jd}|rt| ddnt |df} t |df} t |d|jj|} || d| <t |d|jj|}|| ||d<| |jj|jj|d|jjd|jd}tj || ftjd}tj || ftjd}tj || ftjd}|D]R\}}dt|}tj|dd d | |jd|j}||||||| | 7<|||||| | 7<t|||||||||<q|r|}n|t||}|||d k<t||S) aAggregation Parameters ---------- scores : SlidingWindowFeature Raw (unaggregated) scores. Shape is (num_chunks, num_frames_per_chunk, num_classes). frames : SlidingWindow Frames resolution. warm_up : (float, float) tuple, optional Left/right warm up duration (in seconds). missing : float, optional Value used to replace missing (ie all NaNs) values. skip_average : bool, optional Skip final averaging step. Returns ------- aggregated_scores : SlidingWindowFeature Aggregated scores. Shape is (num_frames, num_classes) r`rerNg?)dtypeFr.)copynan)rvrrrrarr rkroreshapeonesr~ closest_framezerosfloat32isnan nan_to_nummaximumr)rrnrDrrorprrnum_frames_per_chunk num_classesrNhamming_windowwarm_up_window warm_up_left warm_up_right num_framesaggregated_outputoverlapping_chunk_countaggregated_maskrscoremask start_frameaveragerrrrts~            zInference.aggregater/r/c Cs|jjdks Jd|jj\}}}|j}t||d}t||d}t||j|j}||||krItdt |ddd|jdd |jd d |||f}t |j |d|j|jd|d|d|jd } t || S) aTrim left and right warm-up regions Parameters ---------- scores : SlidingWindowFeature (num_chunks, num_frames, num_classes)-shaped scores. warm_up : (float, float) tuple Left/right warm up ratio of chunk duration. Defaults to (0.1, 0.1), i.e. 10% on both sides. Returns ------- trimmed : SlidingWindowFeature (num_chunks, trimmed_num_frames, num_speakers)-shaped scores zLInference.trim expects (num_chunks, num_frames, num_classes)-shaped `scores`rrezTotal `warm_up` is so large (dr-zL% of each chunk) that resulting trimmed scores does not cover a whole step (zs)N)rar r) rvndimrrr~r rr5r6sumrrar) rrDrrrNnum_frames_leftnum_frames_rightnum_frames_stepnew_data new_chunksrrrtrimns* zInference.trim)rNNNFFNrr')r)!rrr__doc__r rrfloatrrkrboolrKr$intrHr1rWrrr\rrr rrrru staticmethodrrrtrrrrrr.s"  [$ #    .  Q  { r)'r5typingrrrrrrrQrkrKtorch.nnr>torch.nn.functional functionalreinopsr"lightning.pytorch.utilities.memoryr pyannote.audio.core.ior pyannote.audio.core.modelr r pyannote.audio.core.taskr pyannote.audio.utils.multi_taskrpyannote.audio.utils.powersetr$pyannote.audio.utils.reproducibilityr pyannote.corerrrrrrrrrs"