o Ti@sZddlmZmZmZmZddlZddlmZmZm Z m Z ddl m Z m Z GdddZdS))AnyCallableOptionalUnionN)GenerationConfigPreTrainedTokenizerPreTrainedTokenizerFastset_seed)SUPPORTED_ARCHITECTURESPreTrainedModelWrapperc@seZdZ    ddedeeefdeee gee fde de d e e d e d e ed dfd dZ  ddeee ejeejeee fdede ee ejfd eee fddZdS)BestOfNSamplerNmodel tokenizerqueries_to_scoreslength_sampler sample_sizeseed n_candidatesgeneration_configreturnc Cs|durt|t|ttfstdt|t|ts)tdt|dt||_||_||_ ||_ ||_ ||_ ||_ dS)a Initialize the sampler for best-of-n generation Args: model (`PreTrainedModelWrapper`): The pretrained model to use for generation tokenizer (`PreTrainedTokenizer` or `PreTrainedTokenizerFast`): Tokenizer associated with the pretrained model queries_to_scores (`Callable[[list[str]], list[float]]`): Callable that takes a list of generated texts and returns the associated reward scores length_sampler (`Any`): Sampler used to sample the length of the generated text sample_size (`int`): Number of samples to generate for each query seed (`int`, *optional*): Random seed used to control generation n_candidates (`int`): Number of candidates to return for each query generation_config (`GenerationConfig`, *optional*): Generation config passed to the underlying model's `generate` method. See `GenerationConfig` (https://huggingface.co/docs/transformers/v4.29.1/en/main_classes/text_generation#transformers.GenerationConfig) for more details NzHtokenizer must be a PreTrainedTokenizer or PreTrainedTokenizerFast, got z,model must be a PreTrainedModelWrapper, got z - supported architectures are: )r isinstancerr ValueErrortyper rrrr gen_configrr) selfrrrrrrrrrP/home/ubuntu/.local/lib/python3.10/site-packages/trl/extras/best_of_n_sampler.py__init__s""   zBestOfNSampler.__init__Ttokenized_queryskip_special_tokensdevicec  sd}t|tjr|jdkr|d}n,t|tr?t|d}|tur+t|d}n|tjur8dd|D}ndd|D}g}|D]B}| |j df}|j j | |f||jd||jj|dt|} fd d| |jjD|qC|S) a Generate the best of n samples for input queries Args: tokenized_query (`list[int]` or `torch.Tensor` or `list[torch.Tensor]` or `list[int]`): represents either a single tokenized query (a single tensor or a list of integers) or a batch of tokenized queries (a list of tensors or a list of lists of integers) skip_special_tokens (`bool`): Whether to remove the special tokens from the output device (`str` or `torch.device`, *optional*): The device on which the model will be loaded **generation_kwargs (`dict`, *optional*): Additional keyword arguments passed along to the underlying model's `generate` method. This is used to override generation config Returns: list[list[str]]: A list of lists of generated texts NrrcSsg|]}|dqS)r)reshape).0tensorrrr qsz+BestOfNSampler.generate..cSsg|] }t|dqSr$)torchr(r&)r'queryrrrr)ss)max_new_tokensr)r"csg|]}|qSrr)r'ioutputrrr)s)rr*Tensorndim unsqueezelistrintr(repeatrrgeneratetorrsqueezer batch_decodertopkrindicesappend) rr!r"r#generation_kwargsqueries element_typeresultr+scoresrr.rr6Os8     zBestOfNSampler.generate)rNrN)TN)__name__ __module__ __qualname__r rrrrr3strfloatrr4rrr r*r0boolr#r6rrrrr sF   :" r )typingrrrrr* transformersrrrr modelsr r r rrrrs