o Xi+M@s<dZddlmZddlmZmZmZddlZddl Z ddl m m Z ddlmZddl mZm Z ddlmZmZmZmZeGd d d ZGd d d e jZGd dde jZGddde jZd ddZGddde jZGddde jZGddde jZGddde jZGddde jZ Gddde jZ!dS)!zyModified from https://github.com/openai/whisper/blob/main/whisper/model.py Add EuclideanCodebook & VectorQuantization ) dataclass)IterableOptionalTupleN rearrange)Tensornn)make_non_pad_mask mask_to_bias onnx2torchmerge_tokenized_segmentsc@sbeZdZUdZeed<dZeed<dZeed<dZeed<d Z eed <d Z eed <d Z e ed<dS) ModelConfign_melsi n_audio_ctxi n_audio_state n_audio_head n_audio_layerin_codebook_sizeFuse_sdpaN) __name__ __module__ __qualname__rint__annotations__rrrrrrboolr r E/home/ubuntu/.local/lib/python3.10/site-packages/s3tokenizer/model.pyrs      rc@eZdZdedefddZdS) LayerNormxreturncCsJt||j|jdur|jnd|jdur|jnd|j|jSN) F layer_normfloatnormalized_shapeweightbiasepstypedtypeselfr$r r r!forward-szLayerNorm.forwardNrrrrr2r r r r!r#+r#c@r")Linearr$r%cCs2t||j|j|jdurdS|j|jSr&)r'linearr+tor/r,r0r r r!r29s   zLinear.forwardNr3r r r r!r57r4r5cs2eZdZdededeedeffdd ZZS)Conv1dr$r+r,r%cs.t|||j|durdS||jSr&)super _conv_forwardr7r/)r1r$r+r, __class__r r!r:Cs  zConv1d._conv_forward)rrrrrr: __classcell__r r r;r!r8As  r8'cCs|ddksJt||dd}t| t|d}t|ddtjf|tjddf}tjt|t|gddS)z*Returns sinusoids for positional embeddingrr Ndim) nplogtorchexparangenewaxiscatsincos)lengthchannels max_timescalelog_timescale_incrementinv_timescales scaled_timer r r! sinusoidsIs  rQc sheZdZddededeffdd Z dded eefd d Z dd ed eded eefddZ Z S)MultiHeadAttentionFn_staten_headrcsNt||_t|||_t||dd|_t|||_t|||_||_dS)NF)r,) r9__init__rTr5querykeyvalueoutr)r1rSrTrr;r r!rUVs     zMultiHeadAttention.__init__Nr$maskcCs@||}||}||}|||||\}}|||fSr&)rVrWrX qkv_attentionrY)r1r$rZqkvwvqkr r r!r2`s   zMultiHeadAttention.forwardr\r]r^c Cs||j\}}}||jd}|jg|jdd|jdRdddd|}|jg|jdd|jdR}|jg|jdd|jdRdddd}|js|dddd|}||}|durj||}|}tjjj |dd |j } | |ddddj dd| fS|dddd|}|dusJtjjj||||d d d } | dd|dd|} | dfS) Ngпr?rr r@) start_dimgg?) attn_mask dropout_pscale)shaperTviewpermuterr)rDr functionalsoftmaxr7r/flattendetachscaled_dot_product_attention transpose contiguoussize) r1r\r]r^rZ_Drfr`woutputr r r!r[lsF 4$0 z MultiHeadAttention.qkv_attention)Fr&) rrrrrrUrrr2r[r=r r r;r!rRTs$  rRcsDeZdZdededeffdd Z d dedeefd d ZZ S) ResidualAttentionBlockrSrTrcsZtt|||d|_t||_|d}tt||t t|||_ t||_ dS)Nr) r9rUrRattnr#attn_lnr Sequentialr5GELUmlpmlp_ln)r1rSrTrn_mlpr;r r!rUs  zResidualAttentionBlock.__init__Nr$rZcCs4||j|||dd}||||}|S)N)rZr)ryrzr}r~)r1r$rZr r r!r2szResidualAttentionBlock.forwardr&) rrrrrrUrrr2r=r r r;r!rvsrvcsXeZdZdededededededeffdd Zd ed ed eeeffd dZZ S) AudioEncoderrn_ctxrSrTn_layerstridercspt||_t|d|dd|_tdddd|_|dt|t fddt |D|_ dS)Nrbr ) kernel_sizerpaddingr?positional_embeddingcsg|] }tdqS)rw)rv).0rrrTrSrr r! s z)AudioEncoder.__init__..) r9rUrr8conv1conv2register_bufferrQr ModuleListrangeblocks)r1rrrSrTrrrr;rr!rUs$ zAudioEncoder.__init__r$x_lenr%cCst|d}t|||}|ddd|jd}t|d}t|||}|ddddd}t|d}|ddd}t||j }||j d|j dddf |j }|j D] }|||d}qf||fS)z x : torch.Tensor, shape = (batch_size, n_mels, T) the mel spectrogram of the audio x_len: torch.Tensor, shape = (batch_size,) length of each audio in x r r?rN)r unsqueezer'gelurrrrir r/rrgr7r)r1r$rrZblockr r r!r2s ( zAudioEncoder.forward) rrrrrrUrrr2r=r r r;r!rs"&rcseZdZdZdedeffdd Zededefdd Z ededefd d Z ed d Z ededefddZ ededefddZ ededefddZZS)EuclideanCodebookzCodebook with Euclidean distance (inference-only). Args: dim (int): Dimension. codebook_size (int): Codebook size. rA codebook_sizecs,tt||}||_|d|dS)Nembed)r9rUrDzerosrr)r1rArrr;r r!rUs  zEuclideanCodebook.__init__r$r%cCst|d}|S)Nz... d -> (...) drr0r r r! preprocess zEuclideanCodebook.preprocesscCsZ|j|j}|djdddd|||djddd }|jddj}|S)Nr?r T)keepdimrrar@)rtr7r/powsummaxindices)r1r$rdist embed_indr r r!quantizes zEuclideanCodebook.quantizecCs|j|ddS)Nra)rh)r1rrgr r r!postprocess_embsz!EuclideanCodebook.postprocess_embrcCst||j}|Sr&)r' embeddingrr1rrr r r! dequantizeszEuclideanCodebook.dequantizecCs*|j}||}||}|||}|Sr&)rgrrr)r1r$rgrr r r!encodes    zEuclideanCodebook.encodecCs||}|Sr&)rrr r r!decoderzEuclideanCodebook.decode)rrr__doc__rrUrDinference_moderrrrrrrr=r r r;r!rs  rcsjeZdZdZdedeffdd ZeddZe de d e fd d Z e d e d e fd dZ Z S)VectorQuantizationzVector quantization implementation (inference-only). Args: dim (int): Dimension codebook_size (int): Codebook size rArcs"tt||d|_||_dS)N)rAr)r9rUr _codebookr)r1rArr;r r!rUs  zVectorQuantization.__init__cCs|jjSr&)rrr1r r r!codebook"szVectorQuantization.codebookr$r%cCs$tj|ddd}|j|}|S)Nr?ra)prA)r' normalizer)rr)r1r$embed_inr r r!r&s zVectorQuantization.encodercCs|j|}t|d}|S)Nzb n d -> b d n)rrrrr r r!r,s  zVectorQuantization.decode)rrrrrrUpropertyrrDrrrrr=r r r;r!rs rc seZdZdZefdedeffdd Zdededeeeffd d Z e dededeeeffd d Z e deded ede deeeff ddZeddZdefddZdefddZddZZS) S3TokenizerzbS3 tokenizer implementation (inference-only). Args: config (ModelConfig): Config nameconfigcsht||_||_t|jj|jj|jj|jj|jj |dkr!dnd|jj |_ t |jj|jj |_dS)Nspeech_tokenizer_v1_25hzr?r )r9rUrrrrrrrrrencoderrr quantizer)r1rrr;r r!rU9s   zS3Tokenizer.__init__melmel_lenr%cCs |||Sr&)r)r1rrr r r!r2Is zS3Tokenizer.forwardcCsHd}||k}|r|||||S|||\}}|j|}||fS)ap Quantize mel spectrogram to tokens, with automatic long audio handling. Args: mel: mel spectrogram tensor, shape (batch_size, n_mels, T) mel_len: mel length tensor, shape (batch_size,) Returns: code: quantized tokens, shape (batch_size, T') code_len: token length, shape (batch_size,) i )any_quantize_mixed_batchrrr)r1rr max_frameslong_audio_maskhiddencode_lencoder r r!rLs  zS3Tokenizer.quantizerrc,Cs|d}d}d}d}d} |||} | ||} | | } g} g}g}t|D]}||}||}||}|sr|ddd|f}|}|| krW| |}t|d|f}| ||tj||jd||ddd d q'd}d}||krt || |}|dd||f}|d }|| kr| |}t|d|f}| ||tj||jd||d |dd |d 7}|| 7}||ksz|}|D]}|d |kr|d r||d<qq'| stj |dtj |jdtj |tj |jdfSt | }t |}| ||\}}|j|} i}!t|D]J\}"}|d }|d }|d}| |"d||"f}#|sJtj|#tj |jd}$|$t|#f|!|<q||!vrSg|!|<|!||#qt|D]8}||r|!|}%t|dr||jdkr|d}&nd}&t|%| |&d}'tj|'tj |jd}(|(t|'f|!|<q`tdd|!D})tj ||)tj |jd}*tj |tj |jd}+|!D]\}\}$}|$|*|d|f<||+|<q|*|+fS)a Handle mixed batch with both short and long audio using unified batch processing. Args: mel: mel spectrogram tensor, shape (batch_size, n_mels, T) mel_len: mel length tensor, shape (batch_size,) long_audio_mask: boolean mask for long audio, shape (batch_size,) max_frames: maximum frames for short audio Returns: code: quantized tokens, shape (batch_size, T') code_len: token length, shape (batch_size,) ri>rxN)deviceFr ) batch_idx is_long_audio segment_idxtotal_segmentsTrrr)r/rrrspeech_tokenizer_v12)overlap token_ratecss|]}|dVqdS)r Nr )r code_infor r r! sz4S3Tokenizer._quantize_mixed_batch..)rqritemr'padappendrDtensorrminrlongstackrrr enumeratecpunumpytolistlenhasattrrrrvaluesitems),r1rrrr batch_size sample_rate hop_length window_sizerframes_per_windowframes_per_overlapframes_per_stride all_segmentsall_segments_len segment_infor audio_mel audio_mel_lenrsegmentseg_lenpad_sizestartrendrinfounified_batch_melunified_batch_lensrrcodesresultsseg_idx segment_code code_tensor audio_codesr merged_codesmerged_codes_tensor max_code_len output_codesoutput_codes_lenr r r!rjs                z!S3Tokenizer._quantize_mixed_batchcCst|jSr&)next parametersrrr r r!rszS3Tokenizer.device onnx_pathcCst|dd}|j|dddS)NFTstrict)r load_state_dict)r1rckptr r r!init_from_onnxs zS3Tokenizer.init_from_onnx ckpt_pathcCs"tj|ddd}|j|dddS)NrT) map_locationmmapr)rDloadr)r1rrr r r! init_from_pt"szS3Tokenizer.init_from_ptcCs|D]\}}d|_qdS)NF)named_parameters requires_grad)r1rrparamr r r!freeze&szS3Tokenizer.freeze)rrrrrstrrUrrr2rDrrrrrrrr rr=r r r;r!r3s(    0 r)r>)"r dataclassesrtypingrrrrrBrDtorch.nn.functionalr rjr'einopsrrutilsr r r rrr#r5r8rQModulerRrvrrrrr r r r!s*      =94