o s·¯iø9ã@sÈddlZddlmZddlmmZddlmZmZmZm Z ddl m Z Gdd„de ƒZ Gdd„de ƒZ Gd d „d ejƒZGd d „d ejƒZGd d„dejƒZGdd„dejƒZGdd„dejƒZdS)éN)ÚLSTMÚLinearÚ BatchNorm1dÚ Parameteré)Ú BaseModelc@seZdZdd„ZdS)ÚXUMXcOstdƒ‚)NzXXUMX is broken in torch 2.0, use torch<2.0 with asteroid<0.7 to use it until it's fixed.)Ú RuntimeError)ÚselfÚargsÚkwargs©r úI/home/ubuntu/.local/lib/python3.10/site-packages/asteroid/models/x_umx.pyÚ__init__ sÿz XUMX.__init__N)Ú__name__Ú __module__Ú __qualname__rr r r rr s rcs\eZdZdZ           d‡fd d „ Zdd„Zdd„Zdd„Zdd„Z‡Z S)Ú BrokenXUMXaÐCrossNet-Open-Unmix (X-UMX) for Music Source Separation introduced in [1]. There are two notable contributions with no effect on inference: a) Multi Domain Losses - Considering not only spectrograms but also time signals b) Combination Scheme - Considering possible combinations of output instruments When starting to train X-UMX, you can optionally use the above by setting ``loss_use_multidomain'' and ``loss_combine_sources'' which are both set in conf.yml. Args: sources (list): The list of instruments, e.g., ["bass", "drums", "vocals"], defined in conf.yml. window_length (int): The length in samples of window function to use in STFT. in_chan (int): Number of input channels, should be equal to STFT size and STFT window length in samples. n_hop (int): STFT hop length in samples. hidden_size (int): Hidden size parameter of LSTM layers. nb_channels (int): set number of channels for model (1 for mono (spectral downmix is applied,) 2 for stereo). sample_rate (int): sampling rate of input wavs nb_layers (int): Number of (B)LSTM layers in network. input_mean (torch.tensor): Mean for each frequency bin calculated in advance to normalize the mixture magnitude spectrogram. input_scale (torch.tensor): Standard deviation for each frequency bin calculated in advance to normalize the mixture magnitude spectrogram. max_bin (int): Maximum frequency bin index of the mixture that X-UMX should consider. Set to None to use all frequency bins. bidirectional (bool): whether we use LSTM or BLSTM. spec_power (int): Exponent for spectrogram calculation. return_time_signals (bool): Set to true if you are using a time-domain loss., i.e., applies ISTFT. If you select ``MDL=True'' via conf.yml, this is set as True. References [1] "All for One and One for All: Improving Music Separation by Bridging Networks", Ryosuke Sawata, Stefan Uhlich, Shusuke Takahashi and Yuki Mitsufuji. https://arxiv.org/abs/2010.04228 (and ICASSP 2021) éééééD¬éNTrFc stƒ |¡||_||_||_||_||_||_||_| |_ |dd|_ | r+| |_ n|j |_ ||_ | |_ | durGt | d|j … ¡ ¡} nt |j ¡} | dur`t d| d|j …¡ ¡} nt |j ¡} t|||dd}t| |dkd}t ||¡|_| rƒ|dn|}i}i}i}i}|D]V}t|j ||d||<t|||| dd d ||<t|j ||d ||<t|  ¡ƒ|d  |¡<t|  ¡ƒ|d  |¡<tt |j ¡ ¡ƒ|d |¡<tt |j ¡ ¡ƒ|d |¡<qt |¡|_t |¡|_ t |¡|_!t "|¡|_#t$|j%||dd|_&dS)Nrrgð?T)Ú window_lengthÚn_fftÚn_hopÚcenter)Ú spec_powerÚmono)Únb_binsÚ hidden_sizeÚ nb_channelsFgš™™™™™Ù?)Ú input_sizer!Ú num_layersÚ bidirectionalÚ batch_firstÚdropout)Únb_output_binsr!r"ú input_mean_{}úinput_scale_{}úoutput_mean_{}úoutput_scale_{})ÚwindowrÚ hop_lengthr)'ÚsuperrrÚin_chanrÚsourcesÚ_return_time_signalsr"Ú nb_layersr%r(Úmax_binr!rÚtorchÚ from_numpyÚfloatÚzerosÚonesÚ_STFTÚ _SpectrogramÚnnÚ SequentialÚencoderÚ_InstrumentBackboneEncrÚ_InstrumentBackboneDecrÚcloneÚformatÚ ModuleDictÚ layer_encÚ layer_lstmÚ layer_decÚ ParameterDictÚ mean_scaleÚ_ISTFTr-Údecoder)r r1rr0rr!r"Ú sample_rater3Ú input_meanÚ input_scaler4r%rÚreturn_time_signalsÚstftÚspecÚlstm_hidden_sizeÚsrc_encÚsrc_lstmÚsrc_decrHÚsrc©Ú __class__r rr8sz    ý ú  ýÿÿ    zBrokenXUMX.__init__cCs`| |¡\}}| | ¡¡}| ||¡}|jr*| ddddd¡}| ||¡}||fSd}||fS)aÇModel forward Args: wav (torch.Tensor): waveform tensor. 1D, 2D or 3D tensor, time last. Returns: masked_mixture (torch.Tensor): estimated spectrograms masked by X-UMX's output of shape $(sources, frames, batch_size, channels, bins)$ time_signals (torch.Tensor): estimated time signals of shape $(sources, batch_size, channels, time_length)$ if `return_time_signals` is `True` rrrérN)r>Úforward_maskerrAÚ apply_masksr2ÚpermuterJ)r ÚwavÚmixtureÚangÚ est_masksÚmasked_mixturerPÚ time_signalsr r rÚforward™s   þzBrokenXUMX.forwardc Cst|jj}|dd|j…f}|g}tdt|jƒƒD] }| | ¡¡qt|jƒD],\}}|||j d  |¡7<|||j d  |¡9<|j ||||ƒ||<q't |ƒt|jƒ}d}t|jƒD]\}}|j ||ƒ} |t ||| dgd¡7}qd|t|jƒ}g} |jD]&}|j|||ƒ} | |j d  |¡9} | |j d   |¡7} |  t | ¡¡qŠtj| dd } | S) N.rr)r*gréÿÿÿÿr,r+©Údim)ÚdataÚshaper4ÚrangeÚlenr1ÚappendrAÚ enumeraterHrBrDÚsumrEr5ÚcatrFÚFÚreluÚstack) r Ú input_specÚshapesÚxÚinputsÚirUÚcross_1Úcross_2Ú tmp_lstm_outÚ mask_listÚx_tmpr_r r rrY¶s. zBrokenXUMX.forward_maskercs(t ‡‡fdd„tt|jƒƒDƒ¡}|S)Ncsg|]}ˆˆ|‘qSr r )Ú.0ru©r_r]r rÚ Ýsz*BrokenXUMX.apply_masks..)r5rprhrir1)r r]r_Ú masked_tf_repr r|rrZÜs$zBrokenXUMX.apply_masksc CsN|j|j|j|jdœ}|j|j|jdd|j|j|j |j ddœ }i|¥|¥}|S)z-Arguments needed to re-instantiate the model.)rr0rrKNF) r1r!r"rLrMr4r3r%rrN) rr0rrKr1r!r"r4r3r%r)r Ú fb_configÚ net_configÚ model_argsr r rÚget_model_argsàs,üöÿþzBrokenXUMX.get_model_args) rrrrrrrNNNTrF) rrrÚ__doc__rrbrYrZr‚Ú __classcell__r r rVrrs(*ña&rcó.eZdZdZ  d‡fdd„ Zdd„Z‡ZS) r?a—Encoder structure that maps the mixture magnitude spectrogram to smaller-sized features which are the input for the LSTM layers. Args: nb_bins (int): Number of frequency bins of the mixture. hidden_size (int): Hidden size parameter of LSTM layers. nb_channels (int): set number of channels for model (1 for mono (spectral downmix is applied,) 2 for stereo). rrcs<tƒ ¡||_||_t t|j||ddt|ƒ¡|_dS)NF)Úbias) r/rr4r!r<r=rrÚenc)r r r!r"rVr rr s  þz_InstrumentBackboneEnc.__init__cCsB|\}}}}| | d||j¡¡}| |||j¡}t |¡}|S©Nrc)r‡Úreshaper4r!r5Útanh©r rsrrÚ nb_framesÚ nb_samplesr"Ú_r r rrbs  z_InstrumentBackboneEnc.forward©rr©rrrrƒrrbr„r r rVrr?þs  ür?cr…) r@a‹Decoder structure that maps output of LSTM layers to magnitude estimate of an instrument. Args: nb_output_bins (int): Number of frequency bins of the instrument estimate. hidden_size (int): Hidden size parameter of LSTM layers. nb_channels (int): Number of output bins depending on STFT size. It is generally calculated ``(STFT size) // 2 + 1''. rrc sXtƒ ¡||_t t|d|ddt|ƒt ¡t||j|ddt|j|ƒ¡|_dS)NrF)Ú in_featuresÚ out_featuresr†) r/rr(r<r=rrÚReLUÚdec)r r(r!r"rVr rr-s  ÿ  ùz_InstrumentBackboneDec.__init__cCs:|\}}}}| | d|jd¡¡}| ||||j¡}|Srˆ)r”r‰rgr(r‹r r rrb?s z_InstrumentBackboneDec.forwardrrr r rVrr@"s  ür@có&eZdZd‡fdd„ Zdd„Z‡ZS) r:rrTcs8tt|ƒ ¡tt |¡dd|_||_||_||_ dS)NF)Ú requires_grad) r/r:rrr5Ú hann_windowr-rrr)r rrrrrVr rrGs  z_STFT.__init__c Csf| ¡\}}}| ||d¡}tj||j|j|j|jddddd }| ¡  |||jdddd¡}|S)z€ Input: (nb_samples, nb_channels, nb_timesteps) Output:(nb_samples, nb_channels, nb_bins, nb_frames, 2) rcFTÚreflect)rr.r-rÚ normalizedÚonesidedÚpad_modeÚreturn_complexrr) Úsizer‰r5rOrrr-rÚ contiguousÚview)r rsrr"Ú nb_timestepsÚstft_fr r rrbNs÷ z _STFT.forward©rrT©rrrrrbr„r r rVrr:Fór:cs&eZdZd‡fdd„ Zdd„Z‡ZS)r;rTcstt|ƒ ¡||_||_dS©N)r/r;rrr)r rrrVr rrls z_Spectrogram.__init__cCsŠ| ¡ ¡}t |tdf|tdf¡}| dd¡}| d¡ d¡ |jd¡}|j r;tj |ddd}tj |ddd}|  dddd¡|gS) zÞ Input: complex STFT (nb_samples, nb_channels, nb_bins, nb_frames, 2) Output: Power/Mag Spectrogram and the corresponding phase (nb_frames, nb_samples, nb_channels, nb_bins) rrrrrcg@T)Úkeepdim) ÚdetachrAr5Úatan2ÚEllipsisÚ transposeÚpowrlrrÚmeanr[)r r¡Úphaser r rrbqs  z_Spectrogram.forward)rTr£r r rVrr;ksr;cr•) rIrrTcs*tt|ƒ ¡||_||_||_||_dSr¥)r/rIrr-rr.r)r r-rr.rrVr rr‹s  z_ISTFT.__init__c CsŒ|j\}}}}}|t |¡}|t |¡} tj|| gdd} |  |||||d¡} tj| |j|j|j |j d} |  |||| jd¡} | S)Nrcrdr)rr.r-r) rgr5ÚcosÚsinrprŸÚistftrr.r-r) r rPr^r1ÚbsizeÚchannelsÚfbinsÚframesÚx_rÚx_irsr\r r rrb’sÿz_ISTFT.forwardr¢r£r r rVrrIŠr¤rI)r5Útorch.nnr<Útorch.nn.functionalÚ functionalrnrrrrÚ base_modelsrrrÚModuler?r@r:r;rIr r r rÚs  o$$%