o wiQ @s,ddlmZmZddlmZmZddlmZmZm Z ddl m Z ddl m Z z&ddlmZmZmZdd lmZdd lmZdd lmZdd lmZd ZWneefy_eZddZdZYnwe dZdefddZ GdddeZ!GdddeZ"GdddeZ#GdddeZ$Gddde Z%dS) )ArgumentParser Namespace)AnyOptional)Pipelineget_supported_taskspipeline)logging)BaseTransformersCLICommand)BodyFastAPI HTTPException)APIRoute) BaseModel) JSONResponse)runTcOsdS)N)xyrrZ/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/transformers/commands/serving.pyr "sr Fztransformers/servingargscCs:t|j|jr |jnd|j|j|jd}t||j|j|j S)z~ Factory function used to instantiate serving server from provided command line arguments. Returns: ServeCommand N)taskmodelconfig tokenizerdevice) r rrrrr ServeCommandhostportworkers)rnlprrrserve_command_factory+sr#c@eZdZUdZeed<dS)ServeModelInfoResultz" Expose model information infosN)__name__ __module__ __qualname____doc__dict__annotations__rrrrr%;  r%c@s.eZdZUdZeeed<eeeed<dS)ServeTokenizeResultz Tokenize result model tokens tokens_idsN) r'r(r)r*liststrr,rintrrrrr.Cs  r.c@r$)ServeDeTokenizeResultz! DeTokenize result model textN)r'r(r)r*r2r,rrrrr4Lr-r4c@r$)ServeForwardResultz Forward result model outputN)r'r(r)r*rr,rrrrr6Tr-r6c@seZdZedefddZdedededefdd Z d d Z d d Z e ddde dddfdede fddZe ddde ddde dddfdeede de fddZe dddfddZdS)rparsercCs|jddd}|jdttdd|jdtdd d |jd td d d |jdtddd |jdtdd|jdtdd|jdtdd|jdtddd |jtddS)z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments servezFCLI tool to run inference requests through REST and GraphQL endpoints.)helpz--taskzThe task to run the pipeline on)typechoicesr:z--host localhostz$Interface the server will listen on.)r;defaultr:z--porti"z Port the serving will listen to.z --workersr zNumber of http workersz--modelz%Model's name or path to stored model.)r;r:z--configz,Model's config name or path to stored model.z --tokenizerzTokenizer name to use.z--devicezSIndicate the device to run onto, -1 indicates CPU, >= 0 indicates GPU (default: -1))funcN) add_parser add_argumentr2rr3 set_defaultsr#)r8 serve_parserrrrregister_subcommand]s,z ServeCommand.register_subcommandr rr r!c Cs||_||_||_||_tstdtd|d|tt d|j t t dgdt d|j tt dgdt d |jtt dgdt d |jtt dgdgd d |_dS) NzUsing serve command requires FastAPI and uvicorn. Please install transformers with [serving]: pip install "transformers[serving]". Or install FastAPI and uvicorn separately.zServing model over :/GET)response_modelresponse_classmethodsz /tokenizePOSTz /detokenizez/forwardiX)routestimeout) _pipelinerr r!_serve_dependencies_installed RuntimeErrorloggerinforr model_infor%rtokenizer. detokenizer4forwardr6_app)selfr rr r!rrr__init__|sR zServeCommand.__init__cCst|j|j|j|jddS)N)rr r!)rrXrr r!rYrrrrszServeCommand.runcCstt|jjjdS)N)r&)r%varsrOrrr[rrrrTszServeCommand.model_infoNT)embedF text_input return_idsc Csjz|jj|}|r|jj|}t||dWSt|dWSty4}z tddt|ddd}~ww)z Tokenize the provided input and eventually returns corresponding tokens id: - **text_input**: String to tokenize - **return_ids**: Boolean flags indicating if the tokens have to be converted to their integer mapping. )r/r0)r/rerror status_codedetailN)rOrrUconvert_tokens_to_idsr. Exceptionrr2)rYr^r_ tokens_txtr0errrrUs zServeCommand.tokenizer0skip_special_tokenscleanup_tokenization_spacesc CsPz|jj|||}td|dWSty'}z tddt|ddd}~ww)a: Detokenize the provided tokens ids to readable text: - **tokens_ids**: List of tokens ids - **skip_special_tokens**: Flag indicating to not try to decode special tokens - **cleanup_tokenization_spaces**: Flag indicating to remove all leading/trailing spaces and intermediate ones. ra)rr5r`rbrdN)rOrdecoder4rhrr2)rYr0rkrl decoded_strrjrrrrVs zServeCommand.detokenizec s\t|dkr tggdSz ||}t|dWSty-}z tddt|id}~ww)zF **inputs**: **attention_mask**: **tokens_type_ids**: r)r7 attention)r7r`rcN)lenr6rOrhrr2)rYinputsr7rjrrrrWs    zServeCommand.forward)r'r(r) staticmethodrrErr2r3rZrrTr boolrUr1rVrWrrrrr\s$1(    rN)&argparserrtypingrr pipelinesrrr utilsr rar fastapir rrfastapi.routingrpydanticrstarlette.responsesruvicornrrP ImportErrorAttributeErrorobject get_loggerrRr#r%r.r4r6rrrrrs0