o wiʁ@sddlZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z m Z ddlmZddlmZddlZddlmZddlmZmZmZmZmZddlmZmZd d lmZe d krlddl!Z!erdd l"m#Z#dd l$m%Z%ddl&m'Z'erddl(Z(ddlm)Z)m*Z*m+Z+e,ej-ej.Z/e,ej-ej0ej.dZ1ddiddiddiddiddiddiddidZ2dZ3dd4e25dZ6gdZ7Gdd d Z8e Gd!d"d"Z9d#e fd$d%Z:Gd&d'd'eZ;dS)(N)ArgumentParser Namespace) dataclassfield)Thread)Optional)disable_progress_bars) AutoTokenizerGenerationConfigPreTrainedTokenizerTextIteratorStreamerlogging)is_rich_availableis_torch_available)BaseTransformersCLICommandWindows)Console)Live)Markdown)AutoModelForCausalLMBitsAndBytesConfigPreTrainedModelz .!\"#$%&'()*+,\-/:<=>?@[]^_`{|}~textz5There is a Llama in my lawn, how can I get rid of it?zyWrite a Python function that integrates any Python function f(x) numerically over an arbitrary interval [x_start, x_end].z4How many helicopters can a human eat in one sitting?z4Count to 10 but skip every number ending with an 'e'zWhy aren't birds real?z2Why is it important to eat socks after meditating?z$Which number is larger, 9.9 or 9.11?)llamacode helicopternumbersbirdssocksnumbers2a **TRANSFORMERS CHAT INTERFACE** Chat interface to try out a model. Besides chatting with the model, here are some basic commands: - **!help**: shows all available commands (set generation settings, save chat, etc.) - **!status**: shows the current status of the model and generation settings - **!clear**: clears the current conversation and starts a new one - **!exit**: closes the interface am **TRANSFORMERS CHAT INTERFACE HELP** Full command list: - **!help**: shows this help message - **!clear**: clears the current conversation and starts a new one - **!status**: shows the current status of the model and generation settings - **!example {NAME}**: loads example named `{NAME}` from the config and uses it as the user input. Available example names: `z`, `a%` - **!set {ARG_1}={VALUE_1} {ARG_2}={VALUE_2}** ...: changes the system prompt or generation settings (multiple settings are separated by a space). Accepts the same flags and format as the `generate_flags` CLI argument. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options - **!save {SAVE_NAME} (optional)**: saves the current chat and settings to file by default to `./chat_history/{MODEL_NAME}/chat_{DATETIME}.yaml` or `{SAVE_NAME}` if provided - **!exit**: closes the interface ) )max_new_tokensr!) do_sampleTr#) num_beamsrr$) temperature?r%)top_k2r')top_pr&r))repetition_penaltyr&r*) eos_tokensN eos_token_id) eos_token_idsNr,c@seZdZddeedeefddZdedefdd Zdefd d Zd d Z defddZ dedefddZ dde fddZ dededefddZdS) RichInterfaceN model_name user_namecCs:t|_|dur d|_n||_|durd|_dS||_dS)N assistantuser)r_consoler/r0)selfr/r0r5W/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/commands/chat.py__init__}s  zRichInterface.__init__ output_streamreturnc Csd}|jd|jdt|jddQ}t|D]D\}}|r#|dkr$qtdd|}||7}g}|D]}||| d rG|d q5|d q5t d | d d }| |qWdn1shwY|j|S)zRStream output from a role, and return the generated text after it's done steaming.z [bold blue]:)consolerefresh_per_secondrz <(/*)(\w*)>z\<\1\2\>z``` z z github-dark) code_themeN)r3printr/r enumerateresub splitlinesappend startswithrjoinstripupdate) r4r8rliveioutputslineslinemarkdownr5r5r6 stream_outputs(        zRichInterface.stream_outputcCs$|jd|jd}|j|S)z!Gets user input from the console. [bold red]: )r3inputr0r@)r4rRr5r5r6rRs zRichInterface.inputcCs|jdS)zClears the console.N)r3clear)r4r5r5r6rSszRichInterface.clearrcCs(|jd|jd||jdS)z%Prints a user message to the console.rQz>:[/ bold red] N)r3r@r0)r4rr5r5r6print_user_messagesz RichInterface.print_user_messagecolorcCs&|jd|d||jdS)z,Prints text in a given color to the console.z[bold ]Nr3r@)r4rrUr5r5r6 print_colorzRichInterface.print_colorFminimalcCs&|jt|rtnt|jdS)z'Prints the help message to the console.N)r3r@rHELP_STRING_MINIMAL HELP_STRING)r4rZr5r5r6 print_helprYzRichInterface.print_helpgeneration_config model_kwargscCsJ|jd|d|r|jd||jd||jdS)zFPrints the status of the model and generation settings to the console.z[bold blue]Model: r>z[bold blue]Model kwargs: z [bold blue]NrW)r4r/r^r_r5r5r6 print_statuss zRichInterface.print_status)NN)F)__name__ __module__ __qualname__rstrr7r rPrRrSrTrXboolr]r dictr`r5r5r5r6r.|s *r.c@seZdZUdZedddidZeeed<edddidZ eeed<eddd idZ eeed <ed dd idZ eed <edddidZ eeed<edddidZ eed<edddidZeeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed <ed!dd"idZeed#<eddd$idZeed%<eddd&idZeed'<eddd(idZeeed)<eddd*idZeeed+<ed,dd-idZeed.<ed/dd0idZeed1<ed2d3gd4d5dZeeed6<eddd7idZeed8<eddd9idZeeed:<eddd;idZ eed<<eddd=idZ!eed><ed?d@dAd?gd5dZ"eedB<edddCidZ#eedD<dS)E ChatArgumentsz Arguments for the chat CLI. See the metadata arg for each argument's description -- the medatata will be printed with `transformers chat --help` Nhelpz_Name of the pre-trained model. The positional argument will take precedence if both are passed.)defaultmetadatamodel_name_or_pathzKUsername to display in chat interface. Defaults to the current user's name.r2zSystem prompt. system_promptz./chat_history/zFolder to save chat history. save_folderz"Path to a yaml file with examples. examples_pathFz7Whether to show runtime warnings in the chat interface.verbosezPath to a local generation config file or to a HuggingFace repo containing a `generation_config.json` file. Other generation settings passed as CLI arguments will be applied on top of this generation config.r^r"z%Maximum number of tokens to generate.r!Tz,Whether to sample outputs during generation.r#rz Number of beams for beam search.r$r&z%Temperature parameter for generation.r%r(zValue of k for top-k sampling.r'z Value of p for nucleus sampling.r)zRepetition penalty.r*z\EOS tokens (text format) to stop the generation. If multiple they should be comma separated.r+zQEOS token IDs to stop the generation. If multiple they should be comma separated.r-mainzLSpecific model version to use (can be a branch name, tag name or commit id).model_revisioncpuzDevice to use for inference.deviceautozOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.)rtbfloat16float16float32)rhchoices torch_dtypez2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant)$rarbrc__doc__rrkrrd__annotations__r2rlrmrnrorer^r!intr#r$r%floatr'r)r*r+r-rqrsryrzr{r|r}rrr5r5r5r6rgs  rgargscCst|S)z; Factory function used to chat with a local model. ) ChatCommand)rr5r5r6chat_command_factory0src@seZdZedefddZddZdedefdd Zede fd d Z ed/ded e e de fddZ ed/de e de efddZde e defddZdedededeeeffddZededede e de e deee eff ddZedede d fd!d"Zdeded#effd$d%Zd&e ded'ed(ee ee e ffded)ed*e edee eeeffd+d,Zd-d.Zd S)0rparsercCsTtf}|jd|d}|d}|jdtddd|jdtdd d d |jtd dS) z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments chat)dataclass_typeszPositional argumentsmodel_name_or_path_positionalNzName of the pre-trained model.)typerirhgenerate_flagsaFlags to pass to `generate`, using a space as a separator between flags. Accepts booleans, numbers, and lists of integers, more advanced parameterization should be set through --generation-config. Example: `transformers chat max_new_tokens=100 do_sample=False eos_token_id=[1,2]`. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options*)rrirhnargs)func)rg add_parseradd_argument_group add_argumentrd set_defaultsr)rr chat_parsergroupr5r5r6register_subcommand8s  zChatCommand.register_subcommandcCs||}||_dSN)_handle_deprecated_argsr)r4rr5r5r6r7Us  zChatCommand.__init__rr9c Csd}|jp|j|_|jdurtd|jdurd}tdttD] \}}}t||}||krAd}td|d|d|d tq!|rKtd t |S) z Handles deprecated arguments and their deprecation cycle. To be removed after we fully migrated to the new args. FNzOne of the following must be provided: - The positional argument containing the model repo, e.g. `transformers chat ` - the optional --model_name_or_path argument, containing the model repo (deprecated)TzThe --model_name_or_path argument is deprecated will be removed in v4.54.0. Use the positional argument instead, e.g. `transformers chat `.zThe --a. argument is deprecated will be removed in v4.54.0. There are two alternative solutions to specify this generation option: 1. Pass `--generation-config ` to specify a generation config. 2. Pass `generate` flags through positional arguments, e.g. `transformers chat =`z (Press enter to continue)) rrk ValueErrorwarningswarn FutureWarning_DEPRECATION_MAPgetattrr@rR)r4r has_warningsdeprecated_arg default_valuenew_argvaluer5r5r6rYs:    z#ChatCommand._handle_deprecated_argscCs$tdkr tSttjS)z)Returns the username of the current user.r)platformsystemosgetloginpwdgetpwuidgetuidpw_namer5r5r5r6 get_usernames zChatCommand.get_usernameNfilenamecCsi}t||d<||d<|j}|dur(td}|jd|d}tj||}tjtj |ddt |d }t j ||d d Wdn1sKwYtj |S) z!Saves the chat history to a file.settings chat_historyNz%Y-%m-%d_%H-%M-%Sz/chat_.jsonT)exist_okwr;)indent)varsrmtimestrftimerrpathrGmakedirsdirnameopenjsondumpabspath)rrr output_dictfoldertime_strfr5r5r6 save_chats    zChatCommand.save_chatrlcCs |durg}|Sd|dg}|S)zClears the chat history.Nrrolecontentr5)rlrr5r5r6clear_chat_historys  zChatCommand.clear_chat_historyrcst|dkriSdd|D}dd|D}dd|D}dtdtfdd fd d|D}d d d |D}d|d}|dd}|dd}|dd}|dd}|dd}|dd}zt|}W|Stjyt dw)zUParses the generate flags from the user input into a dictionary of `generate` kwargs.rcSs.i|]}d|ddd|ddqS)"rrr)split).0flagr5r5r6 s.z4ChatCommand.parse_generate_flags..cSs*i|]\}}||dvr|n|qS))truefalse)lowerrkvr5r5r6rscSs"i|] \}}||dkr dn|qS)Nonenullr5rr5r5r6rs"sr9cSs(|dr |dd}|dddS)N-r.r:)rFreplaceisdigit)rr5r5r6 is_numbers  z3ChatCommand.parse_generate_flags..is_numbercs*i|]\}}||sd|dn|qS)rr5rrr5r6rs*z, cSsg|] \}}|d|qS)z: r5rr5r5r6 sz4ChatCommand.parse_generate_flags..{}z"null"rz"true"rz"false"rz"[[z]"rVr:zFailed to convert `generate_flags` into a valid JSON object. `generate_flags` = {generate_flags} Converted JSON string = {generate_flags_string}) lenitemsrdrerGrrloadsJSONDecodeErrorr)r4rgenerate_flags_as_dictgenerate_flags_stringprocessed_generate_flagsr5rr6parse_generate_flagss2         z ChatCommand.parse_generate_flags tokenizermodelc Cs|jdur3t|j}||||j|j\}}|j|j|j|j |j |j |j ||d }|j di|n d|jvrMtj|j}tj|j} t|| }nt|j}||j} |j di| } || fS)zj Returns a GenerationConfig object holding the generation parameters for the CLI command. N) r!r#r$r%r'r)r* pad_token_idr,rr5)r^copydeepcopyparse_eos_tokensr+r-r!r#r$r%r'r)r*rIrrrbasenamer from_pretrainedrr) r4rrrr^rr-deprecated_kwargsrrparsed_generate_flagsr_r5r5r6get_generation_parameterizations0      z+ChatCommand.get_generation_parameterizationr^r+r-cCs||jdur |j}n|j}g}|dur|||d|dur.|dd|dDt|dkr:||j||fS)z:Retrieves the pad token ID and all possible EOS token IDs.N,cSsg|]}t|qSr5)r)rtoken_idr5r5r6rsz0ChatCommand.parse_eos_tokens..r)rr,extendconvert_tokens_to_idsrrrE)rr^r+r-rall_eos_token_idsr5r5r6r s   zChatCommand.parse_eos_tokens model_argsrcCs@|jrtd|j|j|j|jd}|S|jrtdd}|Sd}|S)NT)r}bnb_4bit_compute_dtyperbnb_4bit_use_double_quantbnb_4bit_quant_storage)r|)r}rryrrr|)rquantization_configr5r5r6get_quantization_config&s z#ChatCommand.get_quantization_configrcCstj|j|j|jd}|jdvr|jntt|j}||}|j|j |d|d}t j|jfd|ji|}t|dddurC| |j }||fS)N)revisionrz)rtNrt)rr{ry device_maprrz hf_device_map) r rrrqrzryrtorchrr{rtors)r4rrryrr_rr5r5r6load_model_and_tokenizer:s.  z$ChatCommand.load_model_and_tokenizer user_input interfaceexamplesr_rcCsd}|dkr||j}|n|dkr|n|drKt|dkrK|} t| dkr6| d} nd} |||| } |jd| d d d n|d r|d d } | } | D]} d| vrq|jd| ddd nq^| | } |j di| }|j di|ng|drt|dkr|d}||vr|g}| ||d| d||ddn4d|dt|d}|j|dd n|dkr|j|j||dnd}|jd|ddd |||||fS)z Handles all user commands except for `!exit`. May update the chat history (e.g. reset it) or the generation config (e.g. set a new flag). Tz!clearz!helpz!saverNzChat saved in !green)rrUz!setr;rz(Invalid flag format, missing `=` after `z;`. Please use the format `arg_1=value_1 arg_2=value_2 ...`.red!examplerr2rzExample z* not found in list of available examples: rz!status)r/r^r_F'z/' is not a valid command. Showing help message.r5)rrlrSr]rFrrrrXrHrrIrTrElistkeysr`r)r4rrr r r^r_r valid_command split_inputrnew_generate_flagsrparsed_new_generate_flagsnew_model_kwargs example_name example_errorr5r5r6handle_non_exit_user_commandsUs`          z)ChatCommand.handle_non_exit_user_commandsc Cststdtstd|j}|jdurt}nt|j }t|}Wdn1s.wY|j dur=| }n|j }| |\}}t |ddd}| |||\}} |jsattt|j|d} | ||j} | jdd zj| } | dr| dkrWdS|j| || ||| | d \} } }} | r| d sWqxn| d | d |j| d dd|j}t |}||||d| }t!|j"|d}|#| $|}|%| d|d Wn t&yYdSwqy)NzHYou need to install rich to use the chat interface. (`pip install rich`)zJYou need to install torch to use the chat interface. (`pip install torch`)T)skip_special_tokens skip_prompt)r/r0)rZr z!exit)rrr r r^r_rrr2rpt)return_tensorsadd_generation_prompt)inputsattention_maskstreamerr^)targetkwargsr1)'r ImportErrorrrrnDEFAULT_EXAMPLESryaml safe_loadr2rrr rror set_verbosity_errorrr.rrSrrlr]rRrFrrEapply_chat_templaterrsr ones_likergeneratestartrPrGKeyboardInterrupt)r4rr rr2rrgeneration_streamerr^r_r rrrr r!generation_kwargsthread model_outputr5r5r6runs|            zChatCommand.runr)rarbrc staticmethodrrr7rgrrdrrrrrfrrr rtupler rr rrrrr.rr3r5r5r5r6r7sl+ 6  *  Tr)sd          X[