o i#@shdZddlmZmZmZmZmZmZddlZddl m Z ddl m Z ddl mZmZGddde ZdS) z$(RNN-)Transducer decoder definition.)AnyDictListOptionalTupleUnionN)check_argument_types) AbsDecoder)ExtendedHypothesis HypothesiscseZdZdZ      d0deded ed ed ed ed effdd Zdej fddZ dede ej e ejffddZdej de ej e ej fde ej e ej e ej fffddZdej dej fddZdedeeefde ej e ej e ej fej ffd d!Zd"eeeeefd#e ej e ej fdeeefd$ede ej e ej ej fej ff d%d&Zd'e ej e ej fd(ede ej e ej ffd)d*Z +d1d'e ej e ej fd,ee ej e ej fd-e edee ej e ej ffd.d/ZZS)2TransducerDecodera(RNN-)Transducer decoder module. Args: vocab_size: Output dimension. layers_type: (RNN-)Decoder layers type. num_layers: Number of decoder layers. hidden_size: Number of decoder units per layer. dropout: Dropout rate for decoder layers. dropout_embed: Dropout rate for embedding layer. embed_pad: Embed/Blank symbol ID. lstm@r vocab_sizernn_type num_layers hidden_sizedropout dropout_embed embed_padcstsJ|dvrtd|ttjj||d|_tjj|d|_ |dkr/tjj ntjj tj fddt |D|_tj fddt |D|_||_|_||_||_d |_||_t|j|_dS) N>grur zNot supported: rnn_type=) padding_idxpr csg|] }dddqS)rT) batch_first.0_)dec_netrrZ/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/asr/decoder/transducer_decoder.py 1sz.TransducerDecoder.__init__..csg|] }tjjdqS)r)torchnnDropoutr)rrr"r#7s)r ValueErrorsuper__init__r$r% Embeddingembedr&rLSTMGRU ModuleListrangedecoder dropout_decdlayersdunitsdtypeodim ignore_idblank_idnext parametersdevice)selfrrrrrrr __class__)r!rrr"r*s,   zTransducerDecoder.__init__r;cCs ||_dS)zNSet GPU device to use. Args: device: Device ID. Nr;)r<r;rrr" set_deviceDs zTransducerDecoder.set_device batch_sizereturncCsJtj|j||j|jd}|jdkr!tj|j||j|jd}||fS|dfS)zInitialize decoder states. Args: batch_size: Batch size. Returns: : Initial decoder hidden states. ((N, B, D_dec), (N, B, D_dec)) r?r N)r$zerosr3r4r;r5)r<rAh_nc_nrrr" init_stateMs  zTransducerDecoder.init_statesequencestatecCs|\}}||d\}}t|jD]M}|jdkrB|j|||||d|||dfd\}\|||d<|||d<n|j|||||dd\}|||d<|j||}q|||ffS)aREncode source label sequences. Args: sequence: RNN input sequences. (B, D_emb) state: Decoder hidden states. ((N, B, D_dec), (N, B, D_dec)) Returns: sequence: RNN output sequences. (B, D_dec) (h_next, c_next): Decoder hidden states. (N, B, D_dec), (N, B, D_dec)) rr r)hx)rFsizer0r3r5r1r2)r<rGrHh_prevc_prevh_nextc_nextlayerrrr" rnn_forwardls   zTransducerDecoder.rnn_forwardlabelscCs4||d}|||}|||\}}|S)zEncode source label sequences. Args: labels: Label ID sequences. (B, L) Returns: dec_out: Decoder output sequences. (B, T, U, D_dec) r)rFrJrr,rP)r<rQrF dec_embeddec_outr rrr"forwards zTransducerDecoder.forwardhypcachecCstjd|jdtj|jd}dttt|j}||vr$||\}}n| |}| ||j \}}||f||<|dd||dfS)a_One-step forward hypothesis. Args: hyp: Hypothesis. cache: Pairs of (dec_out, state) for each label sequence. (key) Returns: dec_out: Decoder output sequence. (1, D_dec) new_state: Decoder hidden states. ((N, 1, D_dec), (N, 1, D_dec)) label: Label ID for LM. (1,) )rrr')r5r;r r) r$fullyseqlongr;joinlistmapstrr,rP dec_state)r<rUrVlabel str_labelsrSr^dec_embrrr"scores  zTransducerDecoder.scorehyps dec_statesuse_lmcCst|}g}dg|}t|D]&\}} dttt| j} | |vr)|| ||<q|| | jd| jfq|rct j dd|D|j d} | | | ddd|D} || } || | \}}d}t|D]&}||dur|||}|||f||<|||f|||d<|d 7}qit jd d|Ddd }| |d d|D}|rt j d d|D|j d|d }|||fS||dfS)aOne-step forward hypotheses. Args: hyps: Hypotheses. states: Decoder hidden states. ((N, B, D_dec), (N, B, D_dec)) cache: Pairs of (dec_out, dec_states) for each label sequences. (keys) use_lm: Whether to compute label ID sequences for LM. Returns: dec_out: Decoder output sequences. (B, D_dec) dec_states: Decoder hidden states. ((N, B, D_dec), (N, B, D_dec)) lm_labels: Label ID sequences for LM. (B,) Nr r'cSsg|]}|dgqSrrrrrrr"r#z1TransducerDecoder.batch_score..r?rcSg|]}|dqS)rrgrrr"r#rcSrirrrdrrr"r#rkdimcSrirfrrmrrr"r#rkcSsg|]}|jdqS)r')rX)rhrrr"r#rh)len enumeraterZr[r\r]rXappendr^r$ LongTensorr;create_batch_statesrFrJr,rPr0 select_statecatview)r<rcrdrVre final_batchprocessdoneirUr`rQ p_dec_statesrarS new_statesjrH lm_labelsrrr" batch_scoresB       zTransducerDecoder.batch_scorestatesidxcCsT|ddd||dddf|jdkr'|ddd||dddffSdfS)a+Get specified ID state from decoder hidden states. Args: states: Decoder hidden states. ((N, B, D_dec), (N, B, D_dec)) idx: State ID to extract. Returns: : Decoder hidden state for given ID. ((N, 1, D_dec), (N, 1, D_dec)) rNrr )r5)r<rrrrr"rws *zTransducerDecoder.select_stateNr check_listcCs@tjdd|Ddd|jdkrtjdd|DddfSdfS)a*Create decoder hidden states. Args: states: Decoder hidden states. ((N, B, D_dec), (N, B, D_dec)) new_states: Decoder hidden states. [N x ((1, D_dec), (1, D_dec))] Returns: states: Decoder hidden states. ((N, B, D_dec), (N, B, D_dec)) cSrirlrrsrrr"r##rkz9TransducerDecoder.create_batch_states..rror cSrirfrrrrr"r#$rkN)r$rxr5)r<rrrrrr"rvs  z%TransducerDecoder.create_batch_states)r rrrrr)N)__name__ __module__ __qualname____doc__intr]floatr*r$r;r@rTensorrtensorrFrPrTr rrrbrrr boolrrwrv __classcell__rrr=r"r s*   $     A r )rtypingrrrrrrr$ typeguardrespnet2.asr.decoder.abs_decoderr -espnet2.asr.transducer.beam_search_transducerr r r rrrr"s