o }oiB@stddlmZmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z Gddde eZGdd d e eZdS) )ABCabstractmethod)AnyDictListOptionalTupleN) Hypothesis) NeuralModulec@seZdZdZedejdejdefddZedejdejfdd Z ed ejdejfd d Z dejdejdejfd dZ e ddZ e ddZdS)AbstractRNNTJointa? An abstract RNNT Joint framework, which can possibly integrate with GreedyRNNTInfer and BeamRNNTInfer classes. Represents the abstract RNNT Joint network, which accepts the acoustic model and prediction network embeddings in order to compute the joint of the two prior to decoding the output sequence. fgreturncCt)a Compute the joint step of the network after the projection step. Args: f: Output of the Encoder model after projection. A torch.Tensor of shape [B, T, H] g: Output of the Decoder model (Prediction Network) after projection. A torch.Tensor of shape [B, U, H] Returns: Logits / log softmaxed tensor of shape (B, T, U, V + 1). Arbitrary return type, preferably torch.Tensor, but not limited to (e.g., see HatJoint) NotImplementedErrorselfr r r^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/modules/rnnt_abstract.pyjoint_after_projections z(AbstractRNNTJoint.joint_after_projectionencoder_outputcCr)z Project the encoder output to the joint hidden dimension. Args: encoder_output: A torch.Tensor of shape [B, T, D] Returns: A torch.Tensor of shape [B, T, H] r)rrrrrproject_encoder, z!AbstractRNNTJoint.project_encoderprednet_outputcCr)z Project the Prediction Network (Decoder) output to the joint hidden dimension. Args: prednet_output: A torch.Tensor of shape [B, U, D] Returns: A torch.Tensor of shape [B, U, H] r)rrrrrproject_prednet9rz!AbstractRNNTJoint.project_prednetcCs|||||S)a= Compute the joint step of the network. Here, B = Batch size T = Acoustic model timesteps U = Target sequence length H1, H2 = Hidden dimensions of the Encoder / Decoder respectively H = Hidden dimension of the Joint hidden step. V = Vocabulary size of the Decoder (excluding the RNNT blank token). NOTE: The implementation of this model is slightly modified from the original paper. The original paper proposes the following steps : (enc, dec) -> Expand + Concat + Sum [B, T, U, H1+H2] -> Forward through joint hidden [B, T, U, H] -- *1 *1 -> Forward through joint final [B, T, U, V + 1]. We instead split the joint hidden into joint_hidden_enc and joint_hidden_dec and act as follows: enc -> Forward through joint_hidden_enc -> Expand [B, T, 1, H] -- *1 dec -> Forward through joint_hidden_dec -> Expand [B, 1, U, H] -- *2 (*1, *2) -> Sum [B, T, U, H] -> Forward through joint final [B, T, U, V + 1]. Args: f: Output of the Encoder model. A torch.Tensor of shape [B, T, H1] g: Output of the Decoder model. A torch.Tensor of shape [B, U, H2] Returns: Logits / log softmaxed tensor of shape (B, T, U, V + 1). )rrrrrrrjointFszAbstractRNNTJoint.jointcCrNrrrrrnum_classes_with_blankfz(AbstractRNNTJoint.num_classes_with_blankcCrrrrrrrnum_extra_outputsjr z#AbstractRNNTJoint.num_extra_outputsN)__name__ __module__ __qualname____doc__rtorchTensorrrrrrpropertyrr!rrrrr s    r csneZdZdZfddZe    d:deejdeejde d ee d e eje ejff d d Z edejd e ejfd dZededee e efd e eje ejejffddZde edee e efd e eje ejejffddZde e ejfddZde ejde d e e ejfddZe d;deejejfeejBd e de dejd eeejejfeejBd eejejfeejBf d!d"Ze d;deejejfeejBd eejejfeejBd#ejd$eeejejfeejBfd%d&Ze d;deejd eejd e dBfd'd(Zed)eejejfeejBd eejejfeejBfd*d+Zedeejd eeejfd,d-Ze d