o GiU@sTddlmZddlmZddlmZddlZddlZddlZddlmZddl m Z Gdd d ej j Z Gd d d ej j ZGd d d ej j ZGdddej j ZGdddej j ZGdddej j Z  d&dede dededededeedeedefddZ    d'dede ded!ed"ed#eejdeedeedefd$d%ZdS)()List)Optional)TupleN)Fsa) DenseFsaVecc @XeZdZededededejdejf ddZedejde d d d ejffd d Z d S) _GetTotScoresFunctionfsas log_semiringuse_double_scores unused_scoresreturncCs2|j||d}||_||_||_|||S)aCompute the total loglikes of an FsaVec. Args: fsas: The input FsaVec (must have 3 axes) log_semiring: True to use log semiring, false to use tropical use_double_scores: False to use float, i.e., single precision floating point, to compute log likes. True to use double precision. unused_scores: It is used only for backward propagation purpose. It should equal `fsas.scores`. Returns: The total loglike for each FSA in `fsas`. If `use_double_scores==True`, its dtype is `torch.float64`; it is `torch.float32` otherwise. r r )_get_tot_scoresdetachr r r save_for_backward)ctxr r r r tot_scoresr?/home/ubuntu/.local/lib/python3.10/site-packages/k2/autograd.pyforward s z_GetTotScoresFunction.forwardtot_scores_gradNc Cs|j}|j}|j}|j\}|dur8||}t|j|\}}|r*t|j||} nt |j||} ddd| fS| ||} |rDtj } ntj } | |j| |} ddd| fS)a Caution: this backward function uses a slightly indirect approach to compute the gradients. Since the tot_scores are just computed as specific elements of `forward_scores`, the obvious way to get derivatives w.r.t. fsas.scores would be to set gradients w.r.t. the forward scores and then use BackpropGetForwardScores() to do the backprop. But that might be a little slower than what we actually do. What we actually do is to compute the backward scores and use them and the forward scores to compute the posteriors, and let the derivs be the (posterior in FSA * loss_deriv w.r.t. that FSA's tot_prob). The result is the same, and the underlying C++ code is simpler. (BackpropGetForwardScores() was added in order to compute slightly more difficult objective functions, that depend on the individual arc posteriors). FN) r r r saved_tensors_get_entering_arcs_k2 shortest_patharcs'get_tot_scores_double_tropical_backward&get_tot_scores_float_tropical_backward _get_arc_post"get_tot_scores_double_log_backward!get_tot_scores_float_log_backward) rrr r r scores entering_arcs_ ragged_int scores_gradarc_post bprop_funcrrrbackwardJs*    z_GetTotScoresFunction.backward __name__ __module__ __qualname__ staticmethodrbooltorchTensorrrr*rrrrr  )r c @r) _GetForwardScoresFunctionr r r r rcC2|j||d}||_||_||_|||S)aCompute the forward scores of an FsaVec. Args: fsas: The input FsaVec (must have 3 axes) log_semiring: True to use log semiring, false to use tropical use_double_scores: False to use float, i.e., single precision floating point, to compute log likes. True to use double precision. unused_scores: It is used only for backward propagation purpose. It should equal `fsas.scores`. Returns: The total loglike for each FSA in `fsas`. If `use_double_scores==True`, its dtype is `torch.float64`; it is `torch.float32` otherwise. r)_get_forward_scoresrr r r r)rr r r r forward_scoresrrrrzs z!_GetForwardScoresFunction.forwardforward_scores_gradNc Csr|j}|j}|j}|j\}|rd}n||}|}|}|r$tjntj } | |j ||||||d} ddd| fS)N) state_batchesleaving_arc_batchesr r$r7forward_scores_deriv) r r r rr_get_state_batches_get_leaving_arc_batchesr"backprop_get_forward_scores_double!backprop_get_forward_scores_floatr) rr8r r r r7r$r9r:r)r'rrrr*s4  z"_GetForwardScoresFunction.backwardr+rrrrr4xr3r4c @r) _GetBackwardScoresFunctionr r r r rcCr5)aHCompute the backward scores of an FsaVec. Args: fsas: The input FsaVec (must have 3 axes) log_semiring: True to use log semiring, false to use tropical use_double_scores: False to use float, i.e., single precision floating point, to compute log likes. True to use double precision. unused_scores: It is used only for backward propagation purpose. It should equal `fsas.scores`. Returns: A torch.Tensor with shape equal to (num_states,) r)_get_backward_scoresrr r r r)rr r r r backward_scoresrrrrs z"_GetBackwardScoresFunction.forwardbackward_scores_gradNc Cs\|j}|j}|j}|j\}|}|}|rtjntj}||j |||||d} ddd| fS)N)r9entering_arc_batchesr rBbackward_scores_deriv) r r r rr<_get_entering_arc_batchesr#backprop_get_backward_scores_double"backprop_get_backward_scores_floatr) rrCr r r rBr9rDr)r'rrrr*s,z#_GetBackwardScoresFunction.backwardr+rrrrr@s &r@c@sleZdZededededejdejdejdejfdd Zed ejde d d d ejejejffd d Z d S)_GetArcPostFunctionr r r r r7rBrcCs.|j||d}||_||_||||S)aqCompute the arc-level posteriors of an FsaVec Args: fsas: The input FsaVec (must have 3 axes) log_semiring: True to use log semiring, false to use tropical use_double_scores: False to use float, i.e., single precision floating point, to compute log likes. True to use double precision. unused_scores: It is used only for backward propagation purpose. It should equal `fsas.scores`. forward_scores: The forward scores of the FSA, computed in a differentiable way by fsas.get_forward_scores(); must be provided as an explicit arg for backprop reasons. backward_scores: The backward scores of the FSA, computed in a differentiable way from fsas.get_backward_scores(); must be provided as an explicit arg for backprop reasons. Returns: The per-arc log-posterior for each arc in `fsas`. If `use_double_scores==True`, its dtype is `torch.float64`; it is `torch.float32` otherwise. r)r rr r r)rr r r r r7rBr(rrrrs' z_GetArcPostFunction.forward arc_post_gradNc Cs\|j}|j}|j\}}|rtjntj}|}|}||j ||\} } ddd|| | fS)N) r r rrbackprop_get_arc_post_doublebackprop_get_arc_post_float_get_incoming_arcsrcloner) rrJr r r7rBr) incoming_arcsarc_scores_gradr8rCrrrr*Bs$   z_GetArcPostFunction.backwardr+rrrrrIs& 1rIc@seZdZe  ddededeededededed e j d e j d e e d e e d e j fddZ ede j d eddddddde j e j f fddZdS)_IntersectDensePrunedFunctionNa_fsasb_fsasout_fsa search_beam output_beammin_active_statesmax_active_statesunused_scores_aunused_scores_bseqframe_idx_nameframe_idx_namerc Cst|dksJtj|j|j||||d\} } }t| |d<|jddD]3\}}t|tj r5t || }nt|t j s=J|j tjksEJ|j| ddd\}}t|d||q$|D] \}}t|d||q\| |_||_||| d}| dur|j}||}|j}t |d|}|t |d|}t|d| rJt|d| || dur|dur|j}||}t|d| rJt|d| ||djS)a Intersect array of FSAs on CPU/GPU. Args: a_fsas: Input FsaVec, i.e., `decoding graphs`, one per sequence. It might just be a linear sequence of phones, or might be something more complicated. Must have either `a_fsas.shape[0] == b_fsas.dim0()`, or `a_fsas.shape[0] == 1` in which case the graph is shared. b_fsas: Input FSAs that correspond to neural network output. out_fsa: A list containing ONLY one entry which will be set to the generated FSA on return. We pass it as a list since the return value can only be types of torch.Tensor in the `forward` function. search_beam: Decoding beam, e.g. 20. Smaller is faster, larger is more exact (less pruning). This is the default value; it may be modified by `min_active_states` and `max_active_states`. output_beam: Pruning beam for the output of intersection (vs. best path); equivalent to kaldi's lattice-beam. E.g. 8. max_active_states: Maximum number of FSA states that are allowed to be active on any given frame for any given intersection/composition task. This is advisory, in that it will try not to exceed that but may not always succeed. You can use a very large number if no constraint is needed. min_active_states: Minimum number of FSA states that are allowed to be active on any given frame for any given intersection/composition task. This is advisory, in that it will try not to have fewer than this number active. Set it to zero if there is no constraint. unused_scores_a: It equals to `a_fsas.scores` and its sole purpose is for back propagation. unused_scores_b: It equals to `b_fsas.scores` and its sole purpose is for back propagation. seqframe_idx_name: If set (e.g. to 'seqframe'), an attribute in the output will be created that encodes the sequence-index and the frame-index within that sequence; this is equivalent to a row-index into b_fsas.values, or, equivalently, an element in b_fsas.shape. frame_idx_name: If set (e.g. to 'frame', an attribute in the output will be created that contains the frame-index within the corresponding sequence. Returns: Return `out_fsa[0].scores`. r)rRrSrUrVrWrXrFinclude_scoresaxisneed_value_indexesN)lenrintersect_dense_prunedr dense_fsa_vecrnamed_tensor_attr isinstancer1r2 index_selectk2 RaggedTensordtypeint32indexsetattrnamed_non_tensor_attr arc_map_a arc_map_br scores_dim1shaperow_ids row_splitshasattrr#)rrRrSrTrUrVrWrXrYrZr[r\ ragged_arcrorpnamea_valuevaluer% seqframe_idxnum_colsrrfsa_idx0 frame_idxrrrr`sV=         z%_IntersectDensePrunedFunction.forward out_fsa_gradc Cs|j\}}|j}|j}tj|d|j|jdd}tj|j|j|jdd }t |||t ||| dddddddd||ddf SNrF)rjdevice requires_grad) rrorpr1zerossizerjrrr contiguousr index_addviewrr~a_scoresb_scoresrorpgrad_agrad_brrrr*<  z&_IntersectDensePrunedFunction.backwardNNr,r-r.r/rrrfloatintr1r2rstrrrr*rrrrrQ^sB     rrQc@seZdZe   ddededeedededede j d e j d e e j d e e d e e d e j fddZ ede j d edddde j e j ffddZdS)_IntersectDenseFunctionNrRrSrTrV max_statesmax_arcsrYrZ a_to_b_mapr[r\rc Cst|dksJtj|j|j| |||d\} } }t| |d<|jddD]3\}}t|tj r5t || }nt|t j s=J|j tjksEJ|j| ddd\}}t|d||q$|D] \}}t|d||q\| |_||_|||d}| dur|j}ttttjddd d kr||}ntj||d d }|j}t |d|}|t |d|}t|d| rJt|d| || dur|dur|j}ttttjddd d kr||}ntj||d d }t|d| rJt|d| ||dj S) a?Intersect array of FSAs on CPU/GPU. Args: a_fsas: Input FsaVec, i.e., `decoding graphs`, one per sequence. It might just be a linear sequence of phones, or might be something more complicated. Must have number of FSAs equal to b_fsas.dim0(), if a_to_b_map not specified. b_fsas: Input FSAs that correspond to neural network output. out_fsa: A list containing ONLY one entry which will be set to the generated FSA on return. We pass it as a list since the return value can only be types of torch.Tensor in the `forward` function. output_beam: Pruning beam for the output of intersection (vs. best path); equivalent to kaldi's lattice-beam. E.g. 8. unused_scores_a: It equals to `a_fsas.scores` and its sole purpose is for back propagation. unused_scores_b: It equals to `b_fsas.scores` and its sole purpose is for back propagation. a_to_b_map: Maps from FSA-index in a to FSA-index in b to use for it. If None, then we expect the number of FSAs in a_fsas to equal b_fsas.dim0(). If set, then it should be a Tensor with ndim=1 and dtype=torch.int32, with a_to_b_map.shape[0] equal to the number of FSAs in a_fsas (i.e. a_fsas.shape[0] if len(a_fsas.shape) == 3, else 1); and elements `0 <= i < b_fsas.dim0()`. seqframe_idx_name: If set (e.g. to 'seqframe'), an attribute in the output will be created that encodes the sequence-index and the frame-index within that sequence; this is equivalent to a row-index into b_fsas.values, or, equivalently, an element in b_fsas.shape. frame_idx_name: If set (e.g. to 'frame', an attribute in the output will be created that contains the frame-index within the corresponding sequence. Returns: Return `out_fsa[0].scores`. r)rRrSrrVrrrFr]r_N.)rfloor) rounding_mode)!rbrintersect_denserrdrrerfr1r2rgrhrirjrkrlrmrnrorprrqtuplemapr __version__splitdivrrrsrtrur#)rrRrSrTrVrrrYrZrr[r\rvrorprwrxryr%rzr{rrr|r}rrrrsf7     "     "  z_IntersectDenseFunction.forwardr~c Cs|j\}}|j}|j}tj|dtj|jdd}tj|jtj|jdd }t |||t ||| ddddddd||dddf Sr) rrorpr1rrfloat32rrrrrrrrrrrr*orz _IntersectDenseFunction.backward)NNNrrrrrrsD     urrRrSrUrVrWrXr[r\rc Csltjjdkr|jdksJ|j|jjddksJdg}t ||||||||j|j|| |dS)aIntersect array of FSAs on CPU/GPU. Caution: `a_fsas` MUST be arc sorted. Args: a_fsas: Input FsaVec, i.e., `decoding graphs`, one per sequence. It might just be a linear sequence of phones, or might be something more complicated. Must have either `a_fsas.shape[0] == b_fsas.dim0()`, or `a_fsas.shape[0] == 1` in which case the graph is shared. b_fsas: Input FSAs that correspond to neural network output. search_beam: Decoding beam, e.g. 20. Smaller is faster, larger is more exact (less pruning). This is the default value; it may be modified by `min_active_states` and `max_active_states`. output_beam: Beam to prune output, similar to lattice-beam in Kaldi. Relative to best path of output. min_active_states: Minimum number of FSA states that are allowed to be active on any given frame for any given intersection/composition task. This is advisory, in that it will try not to have fewer than this number active. Set it to zero if there is no constraint. max_active_states: Maximum number of FSA states that are allowed to be active on any given frame for any given intersection/composition task. This is advisory, in that it will try not to exceed that but may not always succeed. You can use a very large number if no constraint is needed. seqframe_idx_name: If set (e.g. to 'seqframe'), an attribute in the output will be created that encodes the sequence-index and the frame-index within that sequence; this is equivalent to a row-index into b_fsas.values, or, equivalently, an element in b_fsas.shape. frame_idx_name: If set (e.g. to 'frame', an attribute in the output will be created that contains the frame-index within the corresponding sequence. Returns: The result of the intersection. Debugrrr) rversion build_typelabelsminmaxr#rrrQapply) rRrSrUrVrWrXr[r\rTrrrrcs 3 rc@rrrc Csltjjdkr|jdksJ|j|jjddksJdg}t |||||||j|j||| |dS)aIntersect array of FSAs on CPU/GPU. Caution: `a_fsas` MUST be arc sorted. Args: a_fsas: Input FsaVec, i.e., `decoding graphs`, one per sequence. It might just be a linear sequence of phones, or might be something more complicated. Must have `a_fsas.shape[0] == b_fsas.dim0()` if `a_to_b_map` is None. Otherwise, must have `a_fsas.shape[0] == a_to_b_map.shape[0]` b_fsas: Input FSAs that correspond to neural network output. output_beam: Beam to prune output, similar to lattice-beam in Kaldi. Relative to best path of output. max_states: The max number of states to prune the output, mainly to avoid out-of-memory and numerical overflow, default 15,000,000. max_arcs: The max number of arcs to prune the output, mainly to avoid out-of-memory and numerical overflow, default 1073741824(2^30). a_to_b_map: Maps from FSA-index in a to FSA-index in b to use for it. If None, then we expect the number of FSAs in a_fsas to equal b_fsas.dim0(). If set, then it should be a Tensor with ndim=1 and dtype=torch.int32, with a_to_b_map.shape[0] equal to the number of FSAs in a_fsas (i.e. a_fsas.shape[0] if len(a_fsas.shape) == 3, else 1); and elements 0 <= i < b_fsas.dim0(). seqframe_idx_name: If set (e.g. to 'seqframe'), an attribute in the output will be created that encodes the sequence-index and the frame-index within that sequence; this is equivalent to a row-index into b_fsas.values, or, equivalently, an element in b_fsas.shape. frame_idx_name: If set (e.g. to 'frame', an attribute in the output will be created that contains the frame-index within the corresponding sequence. Returns: The result of the intersection (pruned to `output_beam`; this pruning is exact, it uses forward and backward scores. rrrr) rrrrrrr#rrrr) rRrSrVrrrr[r\rTrrrrs 3  rr)rrNNN)typingrrrr1rrhfsarrdrautogradFunctionr r4r@rIrQrrrrrcr2rrrrrsr     ZOGP# P