o i@srddlZddlmZmZmZddlZddlmZddlZddlm Z ddl m Z e ddGdddej ZdS) N)AnyListTuple)nn) make_pad_mask)tablesdecoder_classesOpenAIWhisperDecoderWarpcseZdZdZ    d dededed effd d Zd ej d ej dej dej de ej ej ff ddZ d!dej dej dej de ej de ej e ej ff ddZ ddZdej de edej de ej e effddZZS)"r zvTransformer-based Speech-to-Text Decoder from OpenAI's Whisper Model: URL: https://github.com/openai/whisper smallNF dropout_rate whisper_model download_dir use_padmaskcsft|tvs Jtj||dd}t|j|_|jj j }t j ||_|j~||_dS)Ncpu) download_rootdevice)super__init__whisperavailable_models load_modelcopydeepcopydecoderdecoderstoken_embedding embedding_dimtorchrDropoutdropouttrainr)selfr r rr_model attention_dim __class__U/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/whisper_lid/decoder.pyrs    z!OpenAIWhisperDecoderWarp.__init__hs_padhlens ys_in_pad ys_in_lensreturnc Cs||}}|j||jjd|d}||}||j}|jr7t|dddddf|j }nd}t |jj D]\} } | |||jj |ddd}| t |jj dkr^||}q?|j|}|t|jjj|jdd}||fS)a@Forward decoder. Args: hs_pad: encoded memory, float32 (batch, maxlen_in, feat) hlens: (batch) ys_in_pad: input token ids, int64 (batch, maxlen_out) if input_layer == "embed" input tensor (batch, maxlen_out, #mels) in the other cases ys_in_lens: (batch) Returns: (tuple): tuple containing: x: decoded token score before softmax (batch, maxlen_out, token) if use_output_layer is True, olens: (batch, ) NFT)mask memory_mask is_pad_maskis_pad_memory_maskr)rrpositional_embeddingsizer todtyperrr enumerateblocksr/lenlnr transposeweightfloat) r"r)r*r+r,tgtmemoryxr0layerblockr'r'r(forward+s, "  &  $z OpenAIWhisperDecoderWarp.forwardr>tgt_maskr?cachec Cs|j||jjd|d}||}||j}t|jjD]\}}||||jj d}|t |jjdkr>||}q"|j |}|dddf}|t |jjj|jdd}t j|dd}|dfS)aForward one step. Args: tgt: input token ids, int64 (batch, maxlen_out) tgt_mask: input token mask, (batch, maxlen_out) dtype=torch.uint8 in PyTorch 1.2- dtype=torch.bool in PyTorch 1.2+ (include 1.2) memory: encoded memory, float32 (batch, maxlen_in, feat) cache: cached output list of (batch, max_time_out-1, size) Returns: y, cache: NN output value and cache per `self.decoders`. y.shape` is (batch, maxlen_out, token) NOTE (Shih-Lun): cache implementation is ignored for now for simplicity & correctness Nr.)r/r)dim)rrr3r4r r5r6r7r8r/r9r:rr;r<r= log_softmax) r"r>rDr?rEr@rArByr'r'r(forward_one_step`s"    $z)OpenAIWhisperDecoderWarp.forward_one_stepcCs6|j|dtd|d|d\}}|d|fS)zScore.rrE)rJ unsqueezeremptysqueeze)r"ysstater@logpr'r'r(scores zOpenAIWhisperDecoderWarp.scorerOstatesxscCs$|j|td|dd\}}|dfS)aScore new token batch. Args: ys (torch.Tensor): torch.int64 prefix tokens (n_batch, ylen). states (List[Any]): Scorer states for prefix tokens. xs (torch.Tensor): The encoder feature that generates ys (n_batch, xlen, n_feat). Returns: tuple[torch.Tensor, List[Any]]: Tuple of batchfied scores for next token with shape of `(n_batch, n_vocab)` and next state list for ys. rNrK)rJrrM)r"rOrSrTrQr'r'r( batch_scoresz$OpenAIWhisperDecoderWarp.batch_score)r r NF)N)__name__ __module__ __qualname____doc__r=strboolrrTensorrrCrrJrRrrU __classcell__r'r'r%r(r s` : ')rtypingrrrrrr*funasr.models.transformer.utils.nets_utilsrfunasr.registerrregisterModuler r'r'r'r(s