o ߥi@sddlZddlmZmZddlZddlmZddlmZddlm Z m Z m Z Gdddej Z Gdd d e ZGd d d e ZGd d d e ZGddde ZGddde ZGddde ZGddde ZdS)N)ListOptional)Tensor)ConstraintStateOrderedConstraintStateUnorderedConstraintStatecsteZdZfddZ  dddZejjddZejjde e d e fd d Z d e fd dZ de fddZZS)Searchcs~t|j|_|j|_|j|_dd| D}dd| D}| |t ||_ td|_d|_d|_dS)NcSi|]\}}||qSr .0keyvaluer r e/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/multi_modal/ofa/generate/search.py z#Search.__init__..cSr r r r r r rrF)super__init__ pad_token_idpad unk_token_idunk eos_token_ideos get_vocabitemsget_added_vocabupdatelen vocab_sizetorchtensor src_lengthssupports_constraintsstop_on_max_len)self tokenizertgt_dictadded __class__r rrs      zSearch.__init__NcCst)aTake a single search step. Args: step: the current search step, starting at 0 lprobs: (bsz x input_beam_size x vocab_size) the model's log-probabilities over the vocabulary at the current step scores: (bsz x input_beam_size x step) the historical model scores of each hypothesis up to this point prev_output_tokens: (bsz x step) the previously generated oputput tokens original_batch_idxs: (bsz) the tensor with the batch indices, in the range [0, bsz) this is useful in case there has been applied a re-ordering and we need to know the original indices Return: A tuple of (scores, indices, beams) where: scores: (bsz x output_beam_size) the scores of the chosen elements; output_beam_size can be larger than input_beam_size, e.g., we may return 2*input_beam_size to account for EOS indices: (bsz x output_beam_size) the indices of the chosen elements beams: (bsz x output_beam_size) the hypothesis ids of the chosen elements, in the range [0, input_beam_size) )NotImplementedError)r(steplprobsscoresprev_output_tokensoriginal_batch_idxsr r rr/$sz Search.stepcCs ||_dSN)r%)r(r%r r rset_src_lengthsEs zSearch.set_src_lengthsbatch_constraints beam_sizecCdS)aWInitialize constraint states for constrained decoding (if supported). Args: batch_constraints: (torch.Tensor, optional) the list of constraints, in packed form beam_size: (int) the beam size Returns: *encoder_out* rearranged according to *new_order* Nr )r(r6r7r r rinit_constraintsIs zSearch.init_constraints batch_idxscCr8)a( Removes constraint states for completed sentences (if supported). This is called from sequence_generator._generate() when sentences are deleted from the batch. Args: batch_idxs: Indices of *sentences* whose constraint state should be *kept*. Nr r(r:r r rprune_sentencesXs zSearch.prune_sentences active_hyposcCr8)a Updates the constraint states by selecting the beam items that are retained. This is called at each time step of sequence_generator._generate() when the set of 2 * {beam_size} candidate hypotheses are reduced to the beam size. Args: active_hypos: (batch size, beam size) list of integers denoting, for each sentence, which beam candidate items should be kept. Nr )r(r=r r rupdate_constraintscs zSearch.update_constraintsNN)__name__ __module__ __qualname__rr/r#jitexportr5rrintr9r<r> __classcell__r r r,rr s  !    r c sTeZdZdZfddZejj  d dede e de e de e fd d Z Z S) BeamSearcha' Beam search strategy. step 1. Calculate top k candidates in model's log-probability under descending order. While k is the minor of `beam_size * 2` and `beam_size * vocabulary_size`. step 2. Modify hypothesis score, relative indices, beam indices for the final result. cst|d|_dSr4)rrconstraint_states)r(r*r,r rrzs  zBeamSearch.__init__Nr/r1r2r3c Cs|\}}}|dkr|dddd|ddf}n|dus"J||dddd|dfd}tj||dt|d||dddd} | d} | d} tj| |dd} | |} | | | fS)  Take a single search step. Args: step (`int`): Current step, start with 0. lprobs (`Tensor` with size `(bsz, input_beam_size, vocab_size)`): the model's log-probabilities over the vocabulary at the current step. scores (`Tensor` with size `(bsz, input_beam_size, step - 1)`): Previous sampling scores for each beam. prev_output_tokens (`Tensor`, **optional**. default to `None`): Previous output tokens, no usage in this function, will be deprecated in next version. original_batch_idxs (`Tensor`, **optional**, default to `None`): the tensor with the batch indices, in the range [0, bsz) this is useful in case there has been applied a re-ordering and we need to know the original indices Returns: A tuple of (scores_buf, indices_buf, beams_buf), where: scores_buf (`Tensor` with size `(bsz, output_beam_size)`): The model's log-probabilities over the elements selected to sample from. `output_beam_size` is the minor of `2 * input_beam_size` and `vocab_size - 1`. which cumulates the score before. indices_buf (`Tensor` with size `(bsz, output_beam_size)`): The indices of chosen elements. beams_buf (`Tensor` with size `(bsz, output_beam_size)`): The indices of each beam. rNrrkfloor) rounding_mode) size contiguous unsqueezer#topkviewmindivfmod r(r/r0r1r2r3bszr7r"top_prediction scores_buf indices_buf beams_bufr r rr/~s"#" $    zBeamSearch.stepr? r@rArB__doc__rr#rCrDrErrr/rFr r r,rrGqs rGc sVeZdZdZfddZejjddZejjde de de d e d e f d d Z Z S) PrefixConstrainedBeamSearcha Prefix constrained beam search. step 1. Calculate a mask according to a `prefix_allowed_tokens_fn` function with input of previous hypothesis tokens and indices. step 2. Calculate a candidate set of `lprobs` with `lprobs` and mask produced in step 1. step 3. Just like beam search strategy to generate the hypothesis token. And the difference is the k in top k function is the minor of `beam_size` and `vocab_size -1` cst|||_d|_dSNT)rrprefix_allowed_tokens_fnr')r(r*rar,r rr  z$PrefixConstrainedBeamSearch.__init__c Csz|jd|jd}|dd|f}t|tj }t t ||D]\}\}}d||dd| ||f<q'|S)Nrrr) shaperQrepeatflattentolistr# full_likemathinf enumeratezipra) r(xr2r3r7masksent_isentbatch_ir r r apply_masks z&PrefixConstrainedBeamSearch.apply_maskr/r0r1r2r3c Cs|\}}}|||||d||||||7}|dkr1|dddd|ddf}n|dus7J||dddd|dfd}tj||dt|||dddd} | d} | d} | |} | |} | | | fS)j Take a single search step. Args: step (`int`): Current step, start with 0. lprobs (`Tensor` with size `(bsz, input_beam_size, vocab_size)`): the model's log-probabilities over the vocabulary at the current step. scores (`Tensor` with size `(bsz, input_beam_size, step - 1)`): Previous sampling scores for each beam. prev_output_tokens (`Tensor`, **optional**. default to `None`): Previous output tokens, no usage in this function, will be deprecated in next version. original_batch_idxs (`Tensor`, **optional**, default to `None`): the tensor with the batch indices, in the range [0, bsz) this is useful in case there has been applied a re-ordering and we need to know the original indices Returns: A tuple of (scores_buf, indices_buf, beams_buf), where: scores_buf (`Tensor` with size `(bsz, input_beam_size)`): The model's log-probabilities over the elements selected to sample from. which cumulates the score before. indices_buf (`Tensor` with size `(bsz, input_beam_size)`): The indices of chosen elements. beams_buf (`Tensor` with size `(bsz, input_beam_size)`): The indices of each beam. rrNrrK) rOrqrSrPrQr#rRrTrVrWr r rr/s0" " $    z PrefixConstrainedBeamSearch.step) r@rArBr^rr#rCrDrqrErr/rFr r r,rr_s"   r_cseZdZdZfddZejjdee de fddZ ejjde fd d Z ejjd e fd d Z ejj  dde de dee dee dee f ddZejjde de de deeede de de fddZZS)LexicallyConstrainedBeamSearchaImplements lexically constrained beam search as described in Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation. Post & Vilar, NAACL 2018. https://www.aclweb.org/anthology/N18-1119/ and Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting. Hu et al, NAACL 2019. https://www.aclweb.org/anthology/N19-1090/ This is accomplished by maintaining, for each beam hypothesis, a ConstraintState object (see constraints.py) that tracks which constraints have been generated and using this information to shape the beam for each input sentence. csbt|||_dd|D}dd|D}||t||_d|_ d|_ dS)NcSr r r r r r rr6rz;LexicallyConstrainedBeamSearch.__init__..cSr r r r r r rr7rrT) rrrepresentationrrrr r!r" num_candsr&)r(r)rtr*r+r,r rr3s     z'LexicallyConstrainedBeamSearch.__init__r6r7cs\g|_|D]&}|jdkrt|n |jdkrt||jfddt|DqdS)Nordered unorderedcsg|]}qSr r r iconstraint_stater r MszCLexicallyConstrainedBeamSearch.init_constraints..)rHrtrcreaterappendrange)r(r6r7constraint_tensorr rzrr9@s  z/LexicallyConstrainedBeamSearch.init_constraintsr:csfdd|D_dS)Ncsg|]}j|qSr rHrxr(r rr|Qs zBLexicallyConstrainedBeamSearch.prune_sentences..)rfrHr;r rrr<Os  z.LexicallyConstrainedBeamSearch.prune_sentencesr=csDjr|d}t|D]fdd|Dj<q dSdS)Nrcsg|] }j|qSr rrxr(sentidr rr|Zs zELexicallyConstrainedBeamSearch.update_constraints..)rHrOr)r(r= batch_sizer rrr>Us   z1LexicallyConstrainedBeamSearch.update_constraintsNr/r0r1r2r3c Csd}|j}|\}} } t| d||ddd|_|j} | rb|dkrbg} t| D]\} }t|D]\}}| | |}|jsF| |q4q,t | } | dkrbt j ||| d| |jf<|dkrw|dddd| ddf}n|dus}J||dddd|dfd}t ||d|j}|\}}|| }|| }| s|||fS|dkrt j||| d|dd\}}||d}||d}t j||fdd}t j||fdd}t jd| |d|d}t j||fdd}t j|d| f|d}t j|d| f|d}t j|d| f|d}t| D]5\} }||| || | | || || || \}}}}||| <||| <||| <||j| <q|||fS) a A constrained step builds a large candidates list from the following: - the top 2 * {beam_size} items over the whole beam - for each item in the beam - the top {each_k} (default 1) - all next constraints We then compute the constrained state of each beam item, and assign stripe codes: 0 to the best in each bank, 1 to the 2nd-best, and so on. We then sort by (stripe, score), and truncate the list at 2 * beam size. Args: step: the decoder step lprobs: (batch size, beam size, target vocab) the target-vocab distributions for each item in the beam. Retrun: A tuple of (scores, indices, beams, constraints) where: scores: (batch, output beam size) the scores of the chosen elements indices: (batch, output beam size) the target vocab indices of the chosen elements beams: (batch, output beam size) the 0-indexed hypothesis ids of the chosen elements constraints: (batch, output beam size) the new constraint states rrJrrN)rLdimrdevice)rrOrTrSrurHrjfinishedr~r#r$numelrhrirrPrQrRrVcatarangerdzeroslong step_sentenceclone) r(r/r0r1r2r3each_krrr7r"rHnot_finished_indicessentnosent_constraintsbeamnostateindexrYrZr[r\ top_scores top_indices new_beamsnew_scores_bufnew_indices_buf new_beams_bufstatesindicesbeams new_statesr r rr/_s"     " $           z#LexicallyConstrainedBeamSearch.steprrHr\r[rZcs|j}tD]K\} } tjt| |d} | dkrLt| ftj| |d | d} t| f||  | d} t|| f}|dkrRnq d}fddt |DtjddD|dt| j}d}|||}|jddd \}}||}|||fd d|Dd d }|jd }|||k}t||}t|t|t|d }|d dD] }|sވ|||7}qՇfddt td D}t}d}d}tD]\}}||krd}|}n|d 7}||||||<q|jdd\}}||}||fdd|D|d|j}d|jd|j|fS)aDoes per-sentence processing. Adds all constraints for each hypothesis to the list of candidates; then removes duplicates, sorts, and dynamically stripes across the banks. All tensor inputs are collapsed to those pertaining to a single input sentence. rrrcs"g|] }||qSr )advancerx)r\rHr[r rr|sz@LexicallyConstrainedBeamSearch.step_sentence..cSsg|]}|jqSr )bank)r rr r rr| siT)r descendingcg|]}|qSr r rxrr rr|cSs$tj|dd|ddfddS)z^Rolls a 1d tensor left by 1. [0, 1, 2, 3, 4] becomes [4, 0, 1, 2, 3] rrr)r#rrQ)tr r rroll$s$z:LexicallyConstrainedBeamSearch.step_sentence..rollrNcsg|] }|tdqS)r)r!)r offset)banksr rr|Psrcrr r rxrr rr|dr)rrjr#r$list next_tokensrrrrdrOtakerSrr!tokenssortr" masked_selectpop zeros_likeru)r(r/rr0rHr\r[rZrrrr next_beams next_values cands_sizenum_constraint_tokens MAX_SCOREsort_key sort_values sort_indicesr uniques_maskryrmstripe_offsetsstripescur_bank_countcur_bankrr )rr\rHr[rrs               z,LexicallyConstrainedBeamSearch.step_sentencer?)r@rArBr^rr#rCrDrrrEr9r<r>r/rrrrFr r r,rrs sR     y rscDeZdZdZfddZ  d dedeedeefdd ZZ S) LengthConstrainedBeamSearcha Length constrained beam search for generation. step 1. Build length constraints in model's log-probability. If `min_lens` > `step`, set eos token's score to `-math.inf`, so the generation will not be easily stopped. Otherwise, `max_lens` <= `step`, set eos token's score to `0`, so the generation will be easily stopped. step 2. Using beam search to generate the hypothesis tokens with scores. cs8t|||_||_||_||_t||_d|_dSr`) rr min_len_a min_len_b max_len_a max_len_brGbeamneeds_src_lengths)r(r*rrrrr,r rrys   z$LengthConstrainedBeamSearch.__init__Nr/r2r3cCsh|j|j|j}|j|j|j}tj |||kdd|jf<d|||kdd|jf<|j |||S)rINr) rr%rrrrhrirrr/)r(r/r0r1r2r3min_lensmax_lensr r rr/s "z LengthConstrainedBeamSearch.stepr? r@rArBr^rrErrr/rFr r r,rrns rc sLeZdZdZfddZejj  d dede e de e fdd Z Z S) DiverseBeamSearchzDiverse Beam Search. See "Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models" for details. We only implement the Hamming Diversity penalty here, which performed best in the original paper. cs(t|||_| |_t||_dSr4)rr num_groupsdiversity_strengthrGr)r(r*rrr,r rrs zDiverseBeamSearch.__init__Nr/r2r3c Cs|\}}}||jdkrtdt|dddddf|} ggg} } } t|jD]s} |dd| d|jddf}|dkrV|dd| d|jddfnd}| dkritj|| d|j d}n| }|j |||\}}}| |j| | || || || d|t|| q2tj| dd|d}tj| dd|d}tj| dd|d}|||fS) aj Take a single search step. Args: step (`int`): Current step, start with 0. lprobs (`Tensor` with size `(bsz, input_beam_size, vocab_size)`): the model's log-probabilities over the vocabulary at the current step. scores (`Tensor` with size `(bsz, input_beam_size, step - 1)`): Previous sampling scores for each beam. prev_output_tokens (`Tensor`, **optional**. default to `None`): Previous output tokens, no usage in this function, will be deprecated in next version. original_batch_idxs (`Tensor`, **optional**, default to `None`): the tensor with the batch indices, in the range [0, bsz) this is useful in case there has been applied a re-ordering and we need to know the original indices Returns: A tuple of (scores_buf, indices_buf, beams_buf), where: scores_buf (`Tensor` with size `(bsz, input_beam_size)`): The model's log-probabilities over the elements selected to sample from, which cumulates the score before. indices_buf (`Tensor` with size `(bsz, input_beam_size)`): The indices of chosen elements. beams_buf (`Tensor` with size `(bsz, input_beam_size)`): The indices of each beam. rzIDiverseBeamSearch requires --beam to be divisible by the number of groupsNr)otheralpharJrr)rOr ValueErrorr#rtoraddrQrrPrr/mul_add_r~r scatter_add_onesstackrS)r(r/r0r1r2r3rXr7r" diversity_bufscores_G indices_Gbeams_Gglprobs_gscores_grZr[r\r r rr/s@"&*  zDiverseBeamSearch.stepr?r]r r r,rrs rc sheZdZUdZeed<eed<dfdd Zdd Ze j j dd ed e e d e e fddZZS)Samplingaz Sampling search for generation. 1. Calculate the sample set. 1.1 If `sampling_topk` is not None, chose the candidates which cumulative sum of model's log-probability under descending order is less than `sampling_topk`. 1.2 If `sampling_topp` is not None, chose the top k candidates by model's log-probability under the descending order. 1.3 Chose the whole input set as sampling set. 2. Using multinomial sample strategy to sample candidates from sample set as hypothesis. 3. Modify hypothesis score, relative indices, beam indices for the final result. Attributes: sampling_topk (`int`, **optional**, default to `-1`): The value of k in the sampling strategy of top k. sampling_topp (`float`, **optional**, default to '-1.0'): The value of p The sampling strategy of top p. sampling_topk sampling_topprcst|||_||_dSr4)rrrr)r(r*rrr,r rr"rbzSampling.__init__cCs|}|jdd\}}|jdd}||j}|jdd}|ddddddf}|d|dd|d|d}|} |ddddd| df} |ddddd| df} |ddddd| df} | } | | d}|| fS) aSample among the smallest set of elements whose cumulative probability mass exceeds p. See `"The Curious Case of Neural Text Degeneration" (Holtzman et al., 2019) `_. Args: lprobs: (bsz x input_beam_size x vocab_size) the model's log-probabilities over the vocabulary at the current step Return: A tuple of (trimed_probs, truncated_indices) where: trimed_probs: (bsz x input_beam_size x ?) the model's probabilities over the elements selected to sample from. The width of the third dimension is determined by top-P. truncated_indices: (bsz x input_beam_size x ?) the indices of the chosen elements. T)rrJrNrrr) exp_rcumsumltrclamp_rOscatter_max masked_fill_)r(r0probs sorted_probssorted_indices cumsum_probsrm cumsum_mask last_includedmax_dimtruncated_masktruncated_probstruncated_indices trim_mask trimed_probsr r r _sample_topp's    zSampling._sample_toppNr/r2r3cCs|\}}}|dkr|dddd|ddf}|jdkr(||\} } n|jdkr:||j\}} |} n |} td | } |dkr[tj | |d|dd ||} ntj | ||dddd ||} |dkrx| ||d} tj | d| dd} |  |d} |jdks|jdkrtj | ||dd| ddd} |dkr| ||} n#td| | |d} | tj |dddd|dfd| d| | | fS) rrrNrT) replacementrrJ)rr)rOrPrrrrRrr#emptyr multinomialrSexpandgatherrQlog_squeeze new_zerosrrdr)r(r/r0r1r2r3rXr7r"rrr[rZr\r r rr/Tsd"       $ z Sampling.step)rrr?)r@rArBr^rE__annotations__floatrrr#rCrDrrr/rFr r r,rr s -rcr) DiverseSiblingsSearcha_ Beam search with diverse siblings. See "A Simple, Fast Diverse Decoding Algorithm for Neural Generation" for details. https://arxiv.org/abs/1611.08562 1/ Calculate hypotheses for each beam 2/ Intra-sibling ordering 3/ Rewrite scores 4/ Choose top K hypotheses if diversity_rate == 0 is equivalent to BeamSearch cs t|||_t||_dSr4)rrdiversity_raterGr)r(r*rr,r rrs zDiverseSiblingsSearch.__init__Nr/r2r3cs\}}}t|d|ddd} fddt|D} fddt|D} td| d|j} |dkrG|j ||S |dddd|df dt|D]+} tj dd| ddf|d| | | | | fd | |  || | | q^tj| dd |d}td}tjjd }tjjd }t tj| dd |d| \}}|| }t|D] } || || || <q|||fS) a Take a single search step. Args: step (`int`): Current step, start with 0. lprobs (`Tensor` with size `(bsz, input_beam_size, vocab_size)`): the model's log-probabilities over the vocabulary at the current step. scores (`Tensor` with size `(bsz, input_beam_size, step - 1)`): Previous sampling scores for each beam. prev_output_tokens (`Tensor`, **optional**. default to `None`): Previous output tokens, no usage in this function, will be deprecated in next version. original_batch_idxs (`Tensor`, **optional**, default to `None`): the tensor with the batch indices, in the range [0, bsz) this is useful in case there has been applied a re-ordering and we need to know the original indices Returns: A tuple of (scores_buf, indices_buf, beams_buf), where: final_scores (`Tensor` with size `(bsz, output_beam_size)`): The model's log-probabilities over the elements selected to sample from, which cumulates the score before. `output_beam_size` is the minor of `2 * input_beam_size` and `vocab_size - 1`. final_indices (`Tensor` with size `(bsz, output_beam_size)`): The indices of chosen elements. final_beams (`Tensor` with size `(bsz, ourput_beam_size)`): The indices of each beam. rJrrcsg|] }tdqS)r)r#rrrxr0r rr|sz.DiverseSiblingsSearch.step..csg|] }tjjdqS)r)r# LongTensorrrrxrr rr|srN)outrr)rOrTrSrr#rrrrr/rrQrRfmod_sub_rrrr)r(r/r0r1r2r3rXr7r"rLs_listi_list sibling_scoreryr final_scores final_indices final_beamsr rrr/s>" & ,  zDiverseSiblingsSearch.stepr?rr r r,rrs  r)rhtypingrrr#torch.nnnnrtoken_generation_constraintsrrrModuler rGr_rsrrrrr r r rs"  _NaP@^)