o A@i!@s9ddlmZmZddlZddlZddlZddlZddlZddlZddl m Z m Z m Z m Z ddlmZmZmZmZddlZddlZddlZddlmZddlZddlZddlZddlmZddlmZddlZddlZddlmZm Z dd l!m"Z"dd l m Z ddl#Z#ddl$Z$ddl%Z%dd l&m'Z'ddl(Z(ddl)Z*d Z+d Z,da-d Z.zddl/Z/dZ.WnYzddl0Z0Wn e1yYnwzddl2m3Z3Wne1yddl4m3Z3Ynwzej5Z5Wn e6ye7Z5Ynwze8Z9Wn e:yYnwe;dpdZddde?de@eAdffddZBddZCddZDddZEd e?de?fd!d"ZFd#d$ZGd%e?deHe?eHe?e?fffd&d'ZId(eHe?eHe?e?ffde?fd)d*ZJd+d,ZKd-ZLeGZMzddlNZNeNOeNPd.ZQWnejRSe;d/d0ejRSe;d/d1d.ZQYeQTD]ZUejRSeQeUeLeQeU<ZRejRVeRseWeRqejRSeQd2d3ZXed4d5ZYejRSeQd6d7ZZejR[d8Z\ejRVeZsejRVe\re#]e\eZe^Z_iZ`Gd9d:d:e^Zad;d<Zbd=d>Zcd?d@ZddAe e?e fdBe e?fdCdDZeGdEdFdFe^ZfGdGdHdHe^ZgddIeHe?e?fdBfdJdKZhddLdMZidifdNdOZjdifdPdQZkdifdRdSZldTdUZmddVdWZnddXdYZoddZd[ZpGd\d]d]e^ZqGd^d_d_ejrZseqd`esdaZtdbe de fdcddZuddeejvdfe?dge de?fdhdiZwdeejvde fdjdkZxdle?de?fdmdnZydodpdqdrdsdtdudvdwdxdydzd fd{d|d}d~dddddddxddzdfdddxddzdfdddddfZzdodpdqdrdsdtdudvdwdxddzd fd{dd}d~ddddddxddzdfdddxddzdfdddddfZ{hdZ|dodpdtdd{ddddddddxddzdfdddxddzdfddddfZ}dod{ddddddxddzdfdddxddzdfdddddddfZ~doddddrddudvdwdxddzd fdddd|ddddddxddzdfddddddzdfdfZdZdZdZdZdoddd{ddddddddxddzdfdddddzdfdddddfZdoddddddddzdfddddddddddddddzdfdddddzdfdfZddddddzdfdddddzdfddfZdZdZddddddddzdfdddddzdfddddf ZdZdZdZdeAfddZhdZddddddZdddddZdiiifde?de de fddZddede deddfddZGdddeZddZddZetjefddeAdefdd e?dd d eyd d ddZetjefdde?dddeydd deejvfddZetjefdde?dddeydd deejvfddZddZddZd d!Zd"d#Zd$d%Zetjefd&d'eAdefd(d)edefd*gd+d,d-efd.e?ed/d0efd1e?ed2d0efd3ed4dd5efd6ed7dd5d8d9eyd:d deejvfd;d<Zetjefd=d>eAdefd?d@eAdefdAdBdCedefdDdEdFdGdHdIdJeydKd deejvfdLdMZetjefdNdOe?defddPe?defdQdRdSe?ddTdUeydVd deejvfdWdXZ etjefdYdZe?defd[d\e?d]d^efd_d`e?defdadbe?defdcdde?ded^efdfdgdFdhefdidjdFdhefdkdldFdhefdmdndFdhefdodpdFdhefd*gd+d,d-efd.e?ed/d0efd1e?dqdrefd3ed4dd5efd6ed7dd5dsdteydud deejvfdvdwZetjefddxe?defdydze?d{d^efd|d}e?defd~de?defdde?defdde?dd^ddeydd deejvfddZddZddZddZetjefdde?defdde?defdde?dddeydd ddZetjefdde?defddeAdddeydd deejvfddZetjefdde?defdde?ddddddZetjefdde?ddefddddFdhddeydd ddZdddZetjefdde?defddeAdefdddFddHefdde?defdÐde?defdŐdeAdefdǐdeAdɐd^efdʐdedefd̐de?defdΐdedefdАdedefdҐdedefdԐdeAdefd֐de?dؐd^dِdeydۃd dܐd݄Zetjefdΐdedߐd^efdАdedd^efdҐdedd^efdԐdeAdd^efddeAdd^efdÐde?defd֐de?dؐd^ddeydd ddZddZddZddZetjefd&deAdefdde?defdde?defddedd^efdde?defdde?defdde?defdde?defdde?defdde?defdd dFdhefd d dFdhefd d dFdhefdde?defdddFdhefdddFdhefdddFdhefdejdefdde?defdejddefdddFdhefdd dFdhefd!d"edefd#d$d%eAd&efd'd(d)eAd&efd*d+eAdefd,d-e?defd.d/e?dfd0d1eyd2d deejvfd3d4Zetjefd5d6e?defd7d8e?defd9d:e?defd;deyd?d d@dAZetjefdBdCe?ddDdEeydFd dGdHZetjefddIe?defdJdKe?ddLdMeydNd dOdPZdQdRZetjedSdTeydUd dVdWZetjefd&dXeAdefdAdBdYdZed[efddd\e?dd]d^eyd_d deejvfd`daZetjefd&dbeAdefdAdBdcdZed[efdddde?ddedfeydgd deejvfdhdiZetjefdjdkeAdefddlddmdneydod deejvfdpdqZetjefd&dreAddsdtddudvZetjefd&dweAddxdyddzd{Zetjefd&d|eAdd}d~dddZetjefdjdeAdddeydd deejvfddZetjefd&deAdddeydd ddZetjefd&deAdddeydd ddZetjefdde?ddddddZetjefddddddddeejvfddZetjefddeAdefdde?dddeydd ddZetjefd&deAdddeydd deejvfddZɐddZetjefd&deAdddeydd ddZetjefddeAdddddddZetjdddddZetjefdd`eAdefdde?ddÐdeydŃd dƐdDŽZetjefd&deAdefdɐde?defd*gd+d,d-efd.e?ed/d0efd1e?ed2d0efd3ed4dd5efd6ed7dd5dːdeyd̓d dΐdτZetjefd&deAdefdѐdeAdӐd^efdԐdeAdd^d֐deyd؃d dِdڄZetjefd&deAdefdѐdeAdӐd^efdԐdeAdd^dېdeyd؃d dݐdބZetjefd5de?defdde?ddddddZetjefdjdeAdefddeAddddeydd deejvfddZetjefdde?defddeAdddeydd deejvfddZetjefd&deAdefdde?ddddddZՐddZde e?fddZאdddddddZؐddZِd d Zetjefd d e?de׃d defdde?dgdddefdde?ddrefdQddddefdDdeddd0efddeAdՐdefdde?ddd5efdde?defdde?defdde?defdde?defdde?defdd dFdhefd d dFdhefd d dFdhefdde?defdddFdhefdddFdhefdddFdhefdejdefdde?defdejddefdddFdhefdd dFdhefdde?dfd d!eyd"d d#d$Zetjefd%d&eAdefdԐd'e?defd(d)e?defd*d+dFdhd,d-dd.d/Zetjefd&d0eAdefd1d2edd3d4dd5d6Z etjefd&d7eAdefd*gd+d,d-efd.e?ed/d0efd1e?ed2d0efd3ed4dd5efd6ed7dd5d8d9eyd:d d;d<Zetjefd&d=eAdd>d?eyd@d dAdBZetjefd&dCeAddDdEddFdGZetjefdHdIe?ddIdJddKdLZetjefd&dMeAddNdOddPdQZetjdRdSddTdUZddWdXZdYdZZd[d\Zd]d^Zetjefd&d_eAdd`daeydbd dcddZetjefddeeAdddfdgddhdiZdjdkZetjefd&dleAddmdneydod dpdqZetjefddreAdddsdteydud dvdwZdxdyZhdzZetjefdAd{d|dd}d~deydegdgdddZhdZetjefdAd{d|dd}ddeydegdgdddZetjefdd;dddefdQddddddefdddddddefdddddddefdDddddddefdddFddHefddFddHefdeAdՐdrefddFddHefdeddd0efdde?ddd5efdAdd|dd}ddeydegdgdddZhdZetjefdAd{d|dd}ddeydegdgdddZetjefdddFddHefdeAdՐdrefdeddd0efdde?ddd5efdAdd|dd}ddeydd deejvfddZetjefdddFddHefdeAdՐdrefdeddd0efdde?ddd5efdAdd|dd}ddeydd deejvfddZetjefdddddÐddĐdńZetjefdƐde?ddȐdeydʃd dːd̄Zetjefd&d&eAdd͐dΐddϐdЄZetjefd&d&eAddѐdҐddӐdԄZdՐdքZetjefd&deAddؐdِddڐdۄZetjdܐdݐddސd߄ZetjdddddZddZddZetjdddddZetjdddddZetjddeydd ddZetjddeydd ddZetjdddddZetjefd&deAddddddZetjefdddFddHefdAd.de?defdd1de?defd d d eAdd d dddZddZddZ etjefdAddFddHdddddZ etjefdddFddHefdAd.de?defdd1de?defdddFddHefdd dFd!dHefd"d#dd$d%dd&d'Z etjefd&d(eAdd)d*dd+d,Z etjefdddFddHd-d*diifd.d/Z etjd0d1dd2d3Zetjd4d5eyd6d deejvfd7d8Zetjd9d:eyd;d deejvfd<d=ZetjefdddFd>dHd?d@ddAdBZetjdCdDddEdFZetjefdHdIe?ddGdHddIdJZetjdKdLddMdNZetjefdddFdOdHdPdQeydRd dSdTZetjefdd;dUe?dVd^dWdXeydYd deejvfdZd[Zetjefdjd\eAdefd]d^eAdefd_d`eAdddadbeydcd deejvfdddeZetjefdfdge?defd1dhedefdidjdFd dkdldmeydnd deejvfdodpZetjefd&dqeAdefdΐdedefdАdedefdҐdedefdԐdeAdefdǐdeAdefdʐdedefddre?defddseAdefd̐de?defdddFddHefddte?defdÐde?defdŐdeAddudveydwd dxdyZetjefd&dzeAdefdΐdedefdАdedefdҐdedefdԐdeAdefddeAdefdÐde?dd{d|eyd}d d~dZetjefdde?defdde?ddddddZetjefd&deAdefddeAdefdde?defdde?defdde?defddejdefdde?dddeydd ddZetjefd&deAdefdde?defdJdKe?ddddddZetjefdde?dgeRddeydd ddZetjefd&deAdefdde?ddddddZ ddZ!deejvde de fddZ"etjefd&deAdddeydd ddZ#ddZ$etjefd&deAdddeydd ddZ%etjefd&deAddddddZ&ddZ'etjefd&deAdefd ddedefdAddedefdĐdŐdedefdDdǐdedefddɐdedefddːdedefd d͐deAdefdd1de?defdАdѐdҐdefdӐdԐdeAdefd֐dאdeddِdeydۃd dܐd݄Z(etjefddeAddefd ddedefdAddedefdĐdŐdedefdDdǐdedefddɐdedefddːdedefd d͐deAdefdd1de?defdАdѐdҐdefdӐdԐdeAdefd֐dאdedddeydd ddZ)etjefddeAddefdde?defdDddeAddddeydd ddZ*etjddeydd deejvfddZ+etjefddeAdefdddded[efdd1de?dd^efdAdBddZeAd[dddddZ,etjefd&deAddddddZ-etjefd&deAdefdddded[efdd1de?defdAdBddZdddeyd d d d Z.etjefdd eAddefdddded[efdd1de?defdAdBddZdd deydd ddZ/etjefd&deAddddddZ0etjefd&deAddddddZ1ddZ2etjefd&deAdefddedefdŐdedefdǐdedefdde?defdejd dd!d"eyd#d d$d%Z3d&d'Z4d(d)Z5d*d+Z6etjefd&d,eAdefd(d-edd.d/eyd0d d1d2Z7etjefd&d3eAdefd4d5edefdѐd6edefd7d8e?d9d^d:d;eyd<d d=d>Z8etjefd?d@eAdefdddFddHdAdBddCdDZ9etjefdddFddHdEdBddFdGZ:etjefdHdIe?defdddFdJdHdKdLddMdNZ;etjefd&dOeAddPdQddRdSZ<dTdUZ=dVdWZ>dXdYZ?dZd[Z@d\d]ZAdd^d_ZBd`daZCdbdcZDddedfZEetjefd]dge?defdhdFdidHefdjdFdkdHefdldFdmdHefdndoddefdpdqeAdɐd^efdrdFdsdHdtdueydvd dwdxZFdyZG zGdzd{d{e/jHZIWnYd|d}ZJeKd~krzeJWdSeLeMfyYdSwdS()unicode_literalsprint_functionN)DictListTupleOptional)datedatetime timedeltatimezone)ThreadPoolExecutor)sleep)PIPE)redirect_stdoutredirect_stderr)StringIO)r)Pathzhttps://pypi.orgFT) quote_plusVAST_URLhttps://console.vast.aiLOGLEVELz%(levelname)s - %(message)s)levelformatversionreturn.cCs<|d}t|dkrtd|tjdtdd|DS)N.zInvalid version format: filecs|]}t|VqdSNint).0partr%B/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/vast.py Wz parse_version..)splitlenprintsysstderrtuple)rpartsr%r%r& parse_versionQs  r0cCsTztjgddddd}|j}|dr|ddWS|WSty)YdSw)N)gitdescribez--tagsz --abbrev=0T)capture_outputtextcheckv0.0.0) subprocessrunstdoutstrip startswith Exception)resulttagr%r%r&get_git_versionZs  rAcCs$ztjdWStyYdSw)Nvastair8) importlibmetadatarr>r%r%r%r&get_pip_versionis  rEcCs(z tjdduWStyYdSw)NrBF)rCrDr>r%r%r%r&is_pip_packageps  rFstable_versioncCs>trdtvrtjdtd|Stjd|Sd|S)Nz test.pypi.orgz4 -m pip install --force-reinstall --no-cache-dir -i z vastai==z9 -m pip install --force-reinstall --no-cache-dir vastai==z5git fetch --all --tags --prune && git checkout tags/v)rFPYPI_BASE_PATHr, executable)rGr%r%r&get_update_commandvs  rJcCstrtStSr )rFrErAr%r%r%r&get_local_versionsrK project_namecCsVtd|d}tj|ddid}||jdkr%td|d|j|}|S) Nz/pypi/z/jsonAcceptapplication/jsonheaderszCould not get PyPi Project: z . Response: )rHrequestsgetraise_for_status status_coder>json)rLurlresponse response_datar%r%r&get_project_datas rZ project_datacCs,|d}|s tdt|d}t|S)NinfozCould not get PyPi Projectr)rSr>str)r[ info_data version_datar%r%r&get_pypi_versions r`cCstd}t|}t}t|}t|}||krdStd|d|d}|dvr0tddSt|}tdtj |dddtj tj d }td t d dS) NrBzUpdate available from  to z". Would you like to update [Y/n]: )yzlYou selected no. If you don't want to check for updates each time, update should_check_for_update in vast.pyz Updating...T)shellr5r4r;r-zAUpdate completed successfully! Attempt to run your command again!r) rZr`rKr0inputlowerr+rJr9r:rr,exit) pypi_data pypi_version local_version local_tuple pypi_tupleuser_wants_updateupdate_command_r%r%r&check_for_updates6 rprB)configtempHOMEz.configz.cacherrzgpu_names_cache.json)hoursrq vast_api_key~/.vast_api_keyc@ eZdZdS)ObjectN__name__ __module__ __qualname__r%r%r%r&ryrycCsz8t|}ttdd}t|}tdddddd}t|}||kr-|ks6nt|d|WStyGt|dw)z^Validate that the input value is a valid number for seconds between yesterday and Jan 1, 2100.r7daysi4rz! is not a valid second timestamp.z is not a valid integer.)r"r nowr timestampargparseArgumentTypeError ValueError)valueval yesterday min_secondsmax_date max_secondsr%r%r&validate_secondss   rcCsJt|tr |St|trdd|DSt|tr#dd|DS|S)NcSi|] \}}|t|qSr% strip_stringsr#kr6r%r%r& z!strip_strings..cSg|]}t|qSr%r)r#itemr%r%r& z!strip_strings..) isinstancer]r<dictitemslistrr%r%r&rs   rcCsD|durdSzt|WSty!t|d}t|YSw)Nz%m/%d/%Y)floatrr strptimetimemktime timetuple) date_string date_objectr%r%r&string_to_unix_epochs   rquery date_fieldscCsFi}|D]\}}||vrdd|D}|||<q|||<q|S)zZTakes in a query and date fields to correct and returns query with appropriate epoch datescSrr%)rrr%r%r&r!rz#fix_date_fields..r)rr new_queryfield sub_query new_sub_queryr%r%r&fix_date_fieldss  rc@seZdZddZdS)argumentcOs||_||_dSr )argskwargsselfrrr%r%r&__init__*s zargument.__init__N)r{r|r}rr%r%r%r&r)s rc@s4eZdZddZddZddZddZd d Zd S) hidden_aliasescCs ||_dSr )l)rrr%r%r&r1 zhidden_aliases.__init__cCs t|jSr )iterrrr%r%r&__iter__4rzhidden_aliases.__iter__cCdSNFr%rr%r%r&__bool__7zhidden_aliases.__bool__cCrrr%rr%r%r& __nonzero__:rzhidden_aliases.__nonzero__cCs|j|dSr )rappend)rxr%r%r&r=zhidden_aliases.appendN)r{r|r}rrrrrr%r%r%r&r/s  rrPcCsd}td|jD]v}tj||||d}t}||} tjrht | } t dd| } t d| } | d d} | d d 7<g| ddd d d |  d D} td d | d tdn|| }|jdkr|t||d9}q|S|S)N333333?r)methodrWrPrVz -H '[^']*'rcz (?=\s+-\S+)'z cSsg|]}|qSr%)rstripr#rr%r%r&rLrz http_request.. z \ ig?)rangeretryrRRequestSessionprepare_requestARGScurlcurlifyto_curlresubr)joinr+r<r,rgsendrUrr )verbrreq_urlrPrVtireqsessionprepas_curlsimpler/pprr%r%r& http_request@s(   *     rcCtd||||S)NGETrrrrPrVr%r%r&http_getYrrcCr)NPUTrrr%r%r&http_put\rrcCr)NPOSTrrr%r%r& http_post_rrcCr)NDELETErrr%r%r&http_delbrrcCs8t|d }t|WdS1swYdS)Nr)openrVload) file_pathrr%r%r&load_permissions_from_filefs $rcCttdddS)NT machine_idinternalrshow__instancesrprefixactionparser parsed_argsr%r%r&complete_instance_machinejrrcCr)NTidrrrr%r%r&complete_instancemrrcCsddtddDS)NcSrr%r])r#mr%r%r&rqrz$complete_sshkeys...sshz*.pub)rhomejoinpathglobrr%r%r&complete_sshkeyspsrc@sNeZdZddZddZddZddZd d Zd d d ddZdddZ d S)apwrapcOs^d|vrt|d<tj|i||_|jj|jdd|_g|_d|_g|_ t |_ t |_ dS)Nformatter_classfuncF) MyWideHelpFormatterrArgumentParserr set_defaultsfail_with_help subparsers_subparser_objsadded_help_cmd post_setupsetverbsobjsrr%r%r&rts zapwrap.__init__cOs|jtjtr )r print_helpr,r- SystemExitrakwr%r%r&r szapwrap.fail_with_helpc OsP|ds|jD]}z |j|i|WqtjyYqw|jj|i|S)N parent_only)rSr add_argumentr ArgumentErrorr)rrrrr%r%r&rs  zapwrap.add_argumentcOs4|jdurd|d<d|d<|jj|i||_|jS)Ncommandmetavarzcommand to run. one of:help)rradd_subparsersrr%r%r& subparserss zapwrap.subparserscCs@|r|j||j||d|}|S|j||}|S)N )raddr)rrobjnamer%r%r&get_names    zapwrap.get_namer%Naliasesrcs||jsd_jtddddddfdd}fd d }td kr.helpc s>|jdd}|d\}}}||}rgntg}D]}|d\}}}|||qdvr;td<j|f|d}t |d|t |dj |D]9} |j | j i| j } d} | j d } | d rzt} n| d s| d rt} n| d rt} | rt | d | q]|j|d|S)Nro-z--r"rr' mysignaturemysignature_helprmachinerssh completerr)r{replace partitionr&rrr r! add_parsersetattrrrrrrfr=rendswithrrr ) r  dashed_namerror$r%aliases_transformedrspargtsp myCompleter comparatorr( argumentshelp_rrr%r&inners8        zapwrap.command..innerr7r)rrrr*type)rr(rrArrCr r%r@r&rs&zapwrap.commandcOs|dur tjdd}g}|D]}|r%|d|jvr%|dd|7<q||q|jj|g|Ri|}|jD]}||q;|S)Nr7rr")r,argvrrr parse_argsr)rrErrargv_rrr r%r%r&rFs   zapwrap.parse_argsr ) r{r|r}rr rr!r&rrFr%r%r%r&rss   5rcseZdZfddZZS)r cstj|dddddS)N2r7)widthmax_help_positionindent_increment)superr)rprog __class__r%r&rszMyWideHelpFormatter.__init__)r{r|r}r __classcell__r%r%rOr&r sr z7Use 'vast COMMAND --help' for more info about a command)epilogrdcs"ddfdd|D}|S)a Map over a dict and translate any null string values into ' '. Leave everything else as is. This is needed because you cannot add TableCell objects with only a null string or the client crashes. :param Dict d: dict of item values. :rtype Dict: cSs|dkrdS|S)Nrcr"r%)sr%r%r&translate_nullssz9translate_null_strings_to_blanks..translate_nullscsi|] \}}||qSr%r%rrUr%r&rrz4translate_null_strings_to_blanks..r)rSnew_dr%rVr& translate_null_strings_to_blankss rXrsubpath query_argscCsd}|duri}|jdur|j|d<d}|r/ ddd|D}|jd|d|}n|jd|}|jr\tdt|td td |jd|dd t|td |S) aaCreates the endpoint URL for a given combination of parameters. :param argparse.Namespace args: Namespace with many fields relevant to the endpoint. :param str subpath: added to end of URL to further specify endpoint. :param typing.Dict query_args: specifics such as API key and search parameters that complete the URL. :rtype str: Napi_key&css:|]\}}dj|tt|tr|nt|dVqdS)z{x}={y})rrbN)rrrr]rVdumps)r#rrbr%r%r&r's* zapiurl..z/api/v0r*z query args:rczbase: z + query: )r[rrrWexplainr+)rrYrZr? query_jsonr%r%r&apiurls*   r`cCs i}|jdurd|j|d<|S)zCreates the headers for a given combination of parameters. :param argparse.Namespace args: Namespace with many fields relevant to the endpoint. :rtype Dict: NBearer Authorizationr[)rr?r%r%r& apiheaders-s rdmessagecCs^tjdd|tjd}ddtd|tjD}t|}tjdt|dd|tjd}|S) z Deindent a quoted string. Scans message and finds the smallest number of whitespace characters in any line and removes that many from the start of every line. :param str message: Message to deindent. :rtype str: z *$rc)flagscSsg|] }t|rt|qSr%r*rr%r%r&rBzdeindent..z ^ *(?=[^ ])z^ {,})rr MULTILINEfindallminr]r<)reindentsrr%r%r&deindent9s  rnrID{}NT) cuda_max_goodCUDA{:0.1f}NT)num_gpusN{}xNF)gpu_nameModelrqNT)pcie_bwPCIErtNT)cpu_ghzr|rtNT)cpu_cores_effectivevCPUsrtNTcpu_ramRAMrtcC|dSNr%rr%r%r&Rr) disk_spaceDisk{:.0f}NT) dph_total$/hr{:0.4f}NT)dlperfDLPrtNT)dlperf_per_dphtotalzDLP/${:0.2f}NT)scorerrtNT)driver_versionz NV DriverrqNT)inet_upNet_uprtNT) inet_downNet_downrtNT reliabilityRcC|dSNdr%rr%r%r&r[rdurationMax_DayscCrN@r%rr%r%r&r\r)rmach_idrqNT) verificationstatusrqNT)host_idrrqNT)direct_port_countportsrqNT) geolocationcountryrqNTcCrrr%rr%r%r&rnr)discounted_dph_totalrrNTcCrrr%rr%r%r&rvrcCrrr%rr%r%r&rwr>rdisk_bwhas_avxrrpci_genrzcpu_archrgpu_archverified cuda_versr static_ip cluster_id datacenterrrrr total_flops storage_costrnw_disk_avg_bwnw_disk_max_bwnw_disk_min_bwubuntu_version)rzDisk B/WrtNT) disk_namez Disk NamerqNT)rz $/Gb/Monthz{:.2f}NTcCrrr%rr%r%r&rrcCrrr%rr%r%r&rrcCrrr%rr%r%r&rrcCrrr%rr%r%r&rr)rrrqNT)rz Min BW MiB/srqNT)rz Max BW MiB/srqNT)rz Avg BW MiB/srqNT)rMachinerqNT) actual_statusStatusrqNT)ruNumrwNF)gpu_utilzUtil. %rtNTcCrrr%rr%r%r&rr)rStoragerNT)ssh_hostzSSH AddrrqNT)ssh_portzSSH PortrqNT) image_uuidImagerqNT)rzNet uprtNT)rzNet downrtNT reliability2cCrrr%rr%r%r&rr)labelLabelrqNTz age(hours)rcCrNg @r%rr%r%r&rr) uptime_minsz uptime(mins)rNT)rosubnetSubnetrqNT) node_countNodesrqNT) manager_idz Manager IDrqNT) manager_ipz Manager IPrqNT) machine_idsz Machine ID'srqNT))network_disk_idzNetwork Disk IDrqNT) free_spacezFree Space (GB)rqNT) total_spacezTotal Space (GB)rqNT)r Machine IDrqNT) mount_pointz Mount PointrqNT)) overlay_idz Overlay IDrqNT)r%NamerqNTrrz Cluster IDrqNT)instance_count InstancesrqNT) instancesz Instance IDsrqNTr)rrrqNT)rrrqNTcCrrr%rr%r%r&rrcCrrr%rr%r%r&rr)r VerificationrqNT)rrrqNT)ruz#gpusrqNT)rxrxrqNT)rdiskrqNThostnamerqcCs |ddS)Nr%rr%r%r&r )rdriverrqNT)rreliabrNT)rverirqNT) public_ipaddriprqNT)rgeolocrqNT) num_reportsreportsrqNT)listed_gpu_costzgpuD_$/hrNT) min_bid_pricezgpuI$/hrNT)credit_discount_maxrdiscrNTlisted_inet_up_costz netu_$/TBcCrNir%rr%r%r&rrlisted_inet_down_costz netd_$/TBcCrrr%rr%r%r&r r) gpu_occupancyoccuprqNTr start_timezStart (Date/Time)cCt|dSNz%Y-%m-%d/%H:%Mr fromtimestampstrftimerr%r%r&r'end_timezEnd (Date/Time)cCrrrrr%r%r&r(r)duration_hourszDuration (Hrs)rqNT)maintenance_categoryCategoryrqNT))rrrqNT) first_seenrrqNT)first_locationrrqNT)) ip_addressrrqNT) api_key_idrrqNT) created_atrrqNT) api_routerrqNT)rrrqNT)rzScheduled Job IDrqNT) instance_idz Instance IDrqNT) api_endpointz API EndpointrqNTzStart (Date/Time in UTC)cCrrrrr%r%r&rArzEnd (Date/Time in UTC)cCrrrrr%r%r&rBr)day_of_the_weekzDay of the WeekrqNT)hour_of_the_dayzHour of the Day in UTCrqNT)min_of_the_hourzMinute of the HourrqNT) frequency FrequencyrqNT)) description DescriptionrqNT)quantityQuantityrqNT)rateRaterqNT)amountAmountrqNT)r TimestamprtNT)rDTyperqNT))balanceBalancerqNT)balance_thresholdz Bal. ThldrqNT)balance_threshold_enabledzBal. Thld EnabledrqNT)billaddress_cityCityrqNT)billaddress_countryCountryrqNT)billaddress_line1z Addr Line 1rqNT)billaddress_line2z Addr line 2rqNT)billaddress_zipZiprqNT)billed_expectedzBilled ExpectedrqNT)billed_verifiedz Billed VfyrqNT)billing_creditonlyzBilling CreditonlyrqNT)can_payzCan PayrqNT)creditCreditrNT)emailEmailrqNT)email_verifiedz Email VfyrqNT)fullnamez Full NamerqNT)got_signup_creditzGot Signup CreditrqNT) has_billingz Has BillingrqNT) has_payoutz Has PayoutrqNT)rIdrqNT)last4Last4rqNT) paid_expectedz Paid ExpectedrqNT) paid_verifiedzPaid VfyrqNT)password_resettablezPwd ResettablerqNT) paypal_emailz Paypal EmailrqNT)ssh_keyzSsh KeyrqNT)userUserrqNT)usernameUsernamerqNT)ro)r%NAMErqNT) cloud_typez Cloud TyperqNTcCs|d}|d}dS)a# Accepts two version strings and decides whether a > b, a == b, or a < b. This is meant as a sort function to be used for the driver versions in which only the == operator currently works correctly. Not quite finished... :param str a: :param str b: :return int: rr)r))rba_partsb_partsr%r%r&version_string_sortxs r?>5rrrentedr|rrgpu_ramrrrmin_bidrrzrrexternalrgpu_fracrxrurentabler bw_nvlink cpu_coresrr mobo_namerrrr gpu_mem_bwr compute_caprrr vms_enabled gpu_max_temp inet_up_costrrrr gpu_max_power gpu_total_ramrinet_down_costrrflops_per_dphtotalgpu_display_activer}rrrrRrrrQ)rdisplay_active dlperf_usddph flops_usdrr)rrArOr query_strrescCs|dur|S|dur i}t|tkrd|}|}d}t||}idddddddddd d d d d d d d d ddddddd d dd dd ddddddd}ddd|D}||krytdt|dt||D]\} } } } } | d} || i} | } | | }| |vr| | || } | dkrd| vrt | } | |vrt d | tjd |std!t| d"|d#vrd$d%| d&D} | std't| | | f| std(t| | | f| d)vr|dkrtd*| | vr | | =| |vr|| =q{t| tr$| d+d} | d,} nt| tr8d-d%| D} d.d%| D} | |vrJt| || } | | |<n1| d/ksT| d0krYd1| |<n"| d2ksc| d3krhd4| |<n| d5ksr| d6krwd| |<n| | |<| |vr| || <q{|| | q{|S)7a Basically takes a query string (like the ones in the examples of commands for the search__offers function) and processes it into a dict of URL parameters to be sent to the server. :param str query_str: :param Dict res: :return Dict: Nr"zn([a-zA-Z0-9_]+)( *[=>=gte>gtz<=lte<ltz!=neqz==eq=noteqznot eqnotinznot inin)ninrdrccss|]}d|VqdS)rcN)rrr%r%r&r'szparse_query..z5Unconsumed text. Did you forget to quote your query? z != z,[]rrz?Warning: Unrecognized field: {}, see list of recognized fields.rz6Unknown operator. Did you forget to quote your query? u)rdrccSsg|] }|r|qSr%r<rr%r%r&rrhzparse_query..,z;Value cannot be blank. Did you forget to quote your query? z;Field cannot be blank. Did you forget to quote your query? )r**anyz&Wildcard only makes sense with equals.ro"cSsg|]}|ddqS)ror"r4rr%r%r&r1cSsg|]}|dqS)rkrgrr%r%r&r2trueTrueTfalseFalseFNonenull)rDrrr<rrkrrepr setdefaultrSpopnumeric_versionr+rr,r-r)rr]r4rupdate)rWrXfields field_aliasfield_multiplierpatternoptsop_namesjoinedroprorr6op_namer%r%r& parse_querys                             rrowsrzreplace_spacescCs(dd|D}|g}dd|D}|D]K}g}|||D]?\}} } } } | p*dd} ||d} | dur8d}n | | } | | }|rI|dd }t|}tt|||||<||qq|D]0}g}t|||D]\}}}|\} } } } }|r||}n||}||qkt d |qadS) aBasically takes a set of field names and rows containing the corresponding data and prints a nice tidy table of it. :param list rows: Each row is a dict with keys corresponding to the field names (first element) in the fields tuple. :param Tuple fields: 5-tuple describing a field. First element is field name, second is human readable version, third is format string, fourth is a lambda function run on the data in that field, fifth is a bool determining text justification. True = left justify, False = right justify. Here is an example showing the tuples in action. :rtype None: Example of 5-tuple: ("cpu_ram", "RAM", "{:0.1f}", lambda x: x / 1000, False) cSsg|] \}}}}}|qSr%r%)r#ror%r%r%r&rVrz!display_table..cSrr%rgrr%r%r&rXrcSs|Sr r%rr%r%r&r]szdisplay_table..Nr.r"roz ) rrSrr4r*maxzipljustrjustr+r)rrzrheaderout_rowslengthsinstancerowkeyr%fmtconvrorrTidxoutrfrr%r%r& display_tableJs:           rc@rx) VRLExceptionNrzr%r%r%r&rur~rcCsd}|}d|vr|dd}t|dkr|\}}ntdzt|}d}WnYtd}|dkrB||durBtd|d||fS) z Breaks up a vast-style url in the form instance_id:path and does some basic sanity type-checking. :param url_str: :return: N:z$Invalid VRL (Vast resource locator)./z^(/)?([^/]+(/)?)+$zPath component: z' of VRL is not a valid Unix style path.)r)r*rr"rcompilematch)url_strrpath url_partsvalid_unix_path_regexr%r%r&parse_vast_urlxs      rcCsx|}tj|r t| }|}Wdn1swYd|vr*ttd|ds:ttd ||S)Nz PRIVATE KEYu{ 🐴 Woah, hold on there, partner! That's a *private* SSH key. You need to give the *public* one. It usually starts with 'ssh-rsa', is on a single line, has around 200 or so "base64" characters and ends with some-user@some-where. "Generate public ssh key" would be a good search term if you don't know how to do this. r2a Are you sure that's an SSH public key? Usually it starts with the stanza 'ssh-(keytype)' where the keytype can be things such as rsa, ed25519-sk, or dsa. What you passed me was: {} And welp, that just don't look right. ) osrexistsrreadrrnrfr=r)argstrr5rr%r%r& get_ssh_keys       rrzid of instance to attach torrDr5zssh key to attach to instancez!vastai attach instance_id ssh_keyzaAttach an ssh key to an instance. This will allow you to connect to the instance with the ssh keya6 Attach an ssh key to an instance. This will allow you to connect to the instance with the ssh key. Examples: vast attach 12371 ssh-rsa AAAAB3NzaC1yc2EAAA... vast attach 12371 ssh-rsa $(cat ~/.ssh/id_rsa) The first example attaches the ssh key to instance 12371 )usagerrRcCsNt|j}t|dj|jd}d|i}t||t|d}|t| dS)Nz/instances/{id}/ssh/rr5rPrV) rr5r`rrrrPrTr+rV)rr5rWreq_jsonrr%r%r& attach__sshs rdstz-instance_id:/path to target of copy operationzvastai cancel copy DSTz5Cancel a remote copy in progress, specified by DST ida Use this command to cancel any/all current remote copy operations copying to a specific named instance, given by DST. Examples: vast cancel copy 12371 The first example cancels all copy operations currently copying data into instance 12371 cCt|d}|j}|durtddStd|dd|d}t||t|d}||jd krF|}|d r>td dSt|d dSt|jtd j dit dS)zw Cancel a remote copy in progress, specified by DST id" @param dst: ID of copy instance Target to cancel. /commands/copy_direct/Ninvalid argumentscanceling remote copies to r"me client_iddst_idrrQsuccess\Remote copy canceled - check instance status bar for progress updates (~30 seconds delayed).msg!failed with error {r.status_code}r% r`rr+rrPrTrUrVr4rlocalsrrWrrrrjr%r%r& cancel__copy     rz-instance_id:/path to target of sync operationzvastai cancel sync DSTa Use this command to cancel any/all current remote cloud sync operations copying to a specific named instance, given by DST. Examples: vast cancel sync 12371 The first example cancels all copy operations currently copying data into instance 12371 cCr)z Cancel a remote cloud sync in progress, specified by DST id" @param dst: ID of cloud sync instance Target to cancel. /commands/rclone/Nrrr"rrrrQrrrrr%rrr%r%r& cancel__syncrrcCsttjdS)N%Y-%m-%d)r rr utcrr%r%r%r&default_start_date(srcCsttjtdddS)Nrr)r rr rr rr%r%r%r&default_end_date+srcCstj|tjd}|dS)N)tzr)r rr rr)unix_timestamp utc_datetimer%r%r&convert_timestamp_to_date.s rcC\t|}|dkr dSzt|}d|krdkr|WSWn ty(Ynwtd)zo Accepts an integer string 0-6 or '*' to indicate 'Every day'. Returns 0-6 as int, or None if '*'. riNrz0Day must be 0-6 (0=Sunday) or '*' for every day.r]r<r"rrr)rrdayr%r%r&parse_day_cron_style2   rcCr)zr Accepts an integer string 0-23 or '*' to indicate 'Every hour'. Returns 0-23 as int, or None if '*'. riNrz(Hour must be 0-23 or '*' for every hour.r)rrhourr%r%r&parse_hour_cron_styleBrrrz!id of instance type to change bidz--pricezper machine bid price in $/hourz --schedule)HOURLYDAILYWEEKLYz|try to schedule a command to run hourly, daily, or monthly. Valid values are HOURLY, DAILY, WEEKLY For ex. --schedule DAILY)choicesrz --start_datezTStart date/time in format 'YYYY-MM-DD HH:MM:SS PM' (UTC). Default is now. (optional))rDr+rz --end_datez^End date/time in format 'YYYY-MM-DD HH:MM:SS PM' (UTC). Default is 7 days from now. (optional)z--dayzmDay of week you want scheduled job to run on (0-6, where 0=Sunday) or "*". Default will be 0. For ex. --day 0)rDrr+z--hourzfHour of day you want scheduled job to run on (0-23) or "*" (UTC). Default will be 0. For ex. --hour 16z$vastai change bid id [--price PRICE]z6Change the bid price for a spot/interruptible instancez Change the current bid price of instance id to PRICE. If PRICE is not specified, then a winning bid price is used as the default. cCst|dj|jd}d|jd}|jrtdt||jrBt|j|j |jd}dj|jd}|j|d<t ||||d |jd d St ||t |d }| td |d S)zAlter the bid with id contained in args. :param argparse.Namespace args: should supply all the command-line options :rtype int: z/instances/bid_price/{id}/rrrpricerequest json: z change bidz!/api/v0/instances/bid_price/{id}/rrrNrzPer gpu bid price changed)r`rrrr^r+schedulevalidate_frequency_valuesrradd_scheduled_jobrrPrTrV)rrW json_blob cli_commandrrr%r%r& change__bidRs  rsourcez"id of volume contract being cloneddestz,id of volume offer volume is being copied toz-sz--sizezSize of new volume contract, in GB. Must be greater than or equal to the source volume, and less than or equal to the destination offer.z-dz--disable_compression store_truez+Do not compress volume data before copying.rrz2vastai copy volume [options]zClone an existing volumez Create a new volume with the given offer, by copying the existing volume. Size defaults to the size of the existing volume, but can be increased if there is available space. cCs|j|jd}|jr|j|d<|jrd|d<t|d}|jr&tdt|t||t|d}| |j r7|Std | dS) N)rrsizeTdisable_compressionz/volumes/copy/rr Created. {}) rrrrr`r^r+rrPrTrawrrVrrrWrr%r%r& clone__volume|s   rsrcz>Source location for copy operation (supports multiple formats)z>Target location for copy operation (supports multiple formats)z-iz --identityzLocation of ssh private keyzvastai copy SRC DSTz/Copy directories between instances and/or locala Copies a directory from a source location to a target location. Each of source and destination directories can be either local or remote, subject to appropriate read and write permissions required to carry out the action. Supported location formats: - [instance_id:]path (legacy format, still supported) - C.instance_id:path (container copy format) - cloud_service:path (cloud service format) - cloud_service.cloud_service_id:path (cloud service with ID) - local:path (explicit local path) You should not copy to /root or / as a destination directory, as this can mess up the permissions on your instance ssh folder, breaking future copy operations (as they use ssh authentication) You can see more information about constraints here: https://vast.ai/docs/gpu-instances/data-movement#constraints Examples: vast copy 6003036:/workspace/ 6003038:/workspace/ vast copy C.11824:/data/test local:data/test vast copy local:data/test C.11824:/data/test vast copy drive:/folder/file.txt C.6003036:/workspace/ vast copy s3.101:/data/ C.6003036:/workspace/ The first example copy syncs all files from the absolute directory '/workspace' on instance 6003036 to the directory '/workspace' on instance 6003038. The second example copy syncs files from container 11824 to the local machine using structured syntax. The third example copy syncs files from local to container 11824 using structured syntax. The fourth example copy syncs files from Google Drive to an instance. The fifth example copy syncs files from S3 bucket with id 101 to an instance. c Cst|j\}}t|j\}}|dur|dur td|r!t|dnd|d|r.t|dnd|d||||d}|jrHtdt||dusP|durVt|d }nt|d }t||t|d }| |j d kr7| }|d r |dus|dks|dus|dkr t d} d} |jdur|jn| d} |dus|dkr|d} |d} d| d| d|d| d|d| } t| t j| dd}dS|dus|dkrt jd|dd}|d} |d} d| d| d| d|d|d| } t| t j| dd}dSdS|d rtddS|d d!kr"td"dS|d d#kr/td"dSt|d dSt|jtd$jd%itdS)& Transfer data from one instance to another. @param src: Location of data object to be copied. @param dst: Target to copy object to. Ncopying rrcr"r)rsrc_idrsrc_pathdst_pathrz/commands/rsync/rrrQrlocalz echo $HOMEz /.ssh/id_rsadst_portdst_addrz8sudo rsync -arz -v --progress --rsh=ssh -e 'sudo ssh -i z -p z -o StrictHostKeyChecking=no' z vastai_kaalia@z::rT)rdz mkdir -p src_portsrc_addrz, -o StrictHostKeyChecking=no' vastai_kaalia@zgRemote to Remote copy initiated - check instance status bar for progress updates (~30 seconds delayed).rzsrc_path not supported VMs.zScopy between VM instances does not currently support subpaths (only full disk copy)zdst_path not supported for VMs.rr%)rrrr+r]r^r`rrPrTrUrVr9 getoutputidentityr:r4rr)rrrrrrrWrrhomedir remote_portr remote_addrcmdr?r%r%r&copys`+<   , ((    rz--srcz path to source of object to copyz--dstz path to target of copy operationz /workspace)rrDr+z --instancezid of the instancez --connectionzSid of cloud connection on your account (get from calling 'vastai show connections')z --transferzRtype of transfer, possible options include Instance To Cloud and Cloud To InstancezInstance to Cloud --dry-runz%show what would have been transferredrr --size-onlyz1skip based on size only, not mod-time or checksum--ignore-existingz(skip all files that exist on destination--updatez,skip files that are newer on the destination--delete-excludedz+delete files on dest excluded from transferz]End date/time in format 'YYYY-MM-DD HH:MM:SS PM' (UTC). Default is contract's end. (optional))rDrzovastai cloud copy --src SRC --dst DST --instance INSTANCE_ID -connection CONNECTION_ID --transfer TRANSFER_TYPEz.Copy files/folders to and from cloud providersaZ Copies a directory from a source location to a target location. Each of source and destination directories can be either local or remote, subject to appropriate read and write permissions required to carry out the action. The format for both src and dst is [instance_id:]path. You can find more information about the cloud copy operation here: https://vast.ai/docs/gpu-instances/cloud-sync Examples: vastai show connections ID NAME Cloud Type 1001 test_dir drive 1003 data_dir drive vastai cloud copy --src /folder --dst /workspace --instance 6003036 --connection 1001 --transfer "Instance To Cloud" The example copies all contents of /folder into /workspace on instance 6003036 from gdrive connection 'test_dir'. c Cs"t|d}|jdur|jdurtddSg}|jr|d|jr'|d|jr/|d|jr7|d|j r?|dtd |jd |jd |j d |j d |j |j|j|j |j |j |d }|j rptd t||jrt|j|j|jt|d j|j dddi}t||}||d}|j dkr|r|dd}|durtd|d}|dkrtddSntdd} d} |dd} t||| | d|j | ddStddSt||t|d}||jd krtd!td"dSt|jtd#jd$itdS)%rrNrrrrrrrr")rrrselectedtransferrfr/instances/{id}/rownerrrzinstance to cloudinternet_up_cost_per_tbzInternet upload cost is $zB per TB. Are you sure you want to schedule a cloud backup? (y/n): rbz Cloud backup scheduling aborted.zWarning: Could not retrieve internet upload cost. Proceeding without confirmation. You can use show scheduled-jobs and delete scheduled-job commands to delete scheduled cloud backup job.z cloud copyz/api/v0/commands/rclone/end_dater)rcontract_end_datez1Instance not found. Please check the instance ID.rrQzZCloud Copy Started - check instance status bar for progress updates (~30 seconds delayed).ziWhen the operation is finished you should see 'Cloud Copy Operation Finished' in the instance status bar.rr%) r`rrr+dry_runr size_onlyignore_existingrydelete_excludedr connectionrr^rrrrrrrTrVrfrSrer<rrrPrUr4r) rrWrfrrrrup_costconfirmrrr r%r%r& cloud__copyAsx +     0         rz1instance_id of the container instance to snapshotz--container_registryzEContainer registry to push the snapshot to. Default will be docker.ioz docker.io--repozrepo to push the snapshot toz--docker_login_userz)Username for container registry with repoz--docker_login_passz2Password or token for container registry with repoz--pauseziPause container's processes being executed by the CPU to take snapshot (true/false). Default will be truerozvastai take snapshot INSTANCE_ID --repo REPO --docker_login_user USER --docker_login_pass PASS[--container_registry REGISTRY] [--pause true|false]z[Schedule a snapshot of a running container and push it to your repo in a container registryaC Takes a snapshot of a running container instance and pushes snapshot to the specified repository in container registry. Use pause=true to pause the container during commit (safer but slower), or pause=false to leave it running (faster but may produce a filesystem- // safer snapshot). c Cs|j}|j}|j}|j}|j}|j}td|d|d|||||||d}t|d|d}|jrAtdtt j |dd t ||t |d } | | jd krr| } | d rhtd |d|ddSt| dddSt| jtdjditdS)a Take a container snapshot and push. @param instance_id: instance identifier. @param repo: Docker repository for the snapshot. @param container_registry: Container registry @param docker_login_user: Docker registry username. @param docker_login_pass: Docker registry password/token. @param pause: "true" or "false" to pause the container during commit. zTaking snapshot for instance z and pushing to repo z in container registry )rcontainer_registry personal_repodocker_login_userdocker_login_passpausez/instances/take_snapshot/rz Request JSON:rindentrrQrz;Snapshot request sent successfully. Please check your repo z~ in 5-10 mins. It can take longer than 5-10 mins to push your snapshot image to your repo depending on the size of your image.rz#Unknown error with snapshot requestrNr%)rreporrrrr+r`r^rVr]rrPrTrUrSr4rr) rrrrr6password pause_flagrrWrdatar%r%r&take__snapshots6    rcsrfdd}dkr|dur|dus|dkr$|dur!|dus$|dkr5|dur0|dus7|dSdSdS)NcsFd}dkr |d7}ndkr|d7}ndkr|d7}t|dS)Nrcrz/For HOURLY jobs, day and hour must both be "*".rzHFor DAILY jobs, day must be "*" and hour must have a value between 0-23.rz[For WEEKLY jobs, day must have a value between 0-6 and hour must have a value between 0-23.)r,rg)rr r%r&raise_frequency_errors  z8validate_frequency_values..raise_frequency_errorrrrr%)rrr r r%rr&rs   rc Csht|\}}|jdur|}t||_||krtd|j|j|j} } } t|d} |||||| | | |d } tj | t | d}|j rIt dt | |j dkrct d| d|d |jd |jd dS|j d krst d |j ddS|j dkrtd}|dkr|d}t|d|d} t|| | |j|j| }dSt ddSt d|j d|jdS)Nz*--start_date must be less than --end_date./commands/schedule_job/) rrrrequest_method request_bodyrrr rrrrQz/add_scheduled_job insert: success - Scheduling  job to  from  UTC to  UTCz.add_scheduled_job insert: failed status_code: 7. It could be because you aren't using a valid api_key.iz>Existing scheduled job found. Do you want to update it (y|n)? rbscheduled_job_idrzJob update aborted by the user.z(add_scheduled_job insert: failed error: z. Response body: )convert_dates_to_timestampsrrrrrrr`rRpostrPr^r+rU start_daterer<rfrVupdate_scheduled_jobr4)rrrrr"rr start_timestamp end_timestamprrr schedule_job_urlr#rX user_inputr*r%r%r&rsD      *    rc Cstj|t|d}||jdkr*td|d|d|d|d t||S|jdkr:td |jd |Std |jd t||S) NrrQz/add_scheduled_job update: success - Scheduling r$r%r&r'r(z.add_scheduled_job update: failed status_code: r)r)rRputrPrTrUr+rV)rr1r r-rr#rXr%r%r&r.Ds "   r.--namezname of the api-keyz--permission_filezofile path for json encoded permissions, see https://vast.ai/docs/cli/roles-and-permissions for more informationz --key_paramsz.optional wildcard key params for advanced keysz?vastai create api-key --name NAME --permission_file PERMISSIONSzZCreate a new api-key with restricted permissions. Can be sent to other users and teammatesz In order to create api keys you must understand how permissions must be sent via json format. You can find more information about permissions here: https://vast.ai/docs/cli/roles-and-permissions c Csz(t|d}t|j}t||t|j||jdd}|td | WdSt y:td |jYdSt j jyU}ztd |WYd}~dSd}~wtyl}z td|WYd}~dSd}~ww)N/auth/apikeys/)r% permissions key_paramsrzapi-key created {}z&Error: Permission file '{}' not found.z+Error: Failed to create api-key. Reason: {}zAn unexpected error occurred:)r`rpermission_filerrPr%r7rTr+rrVFileNotFoundErrorrR exceptionsRequestExceptionr>)rrWr6rer%r%r&create__api_keyVs   r=rz*local subnet for cluster, ex: '0.0.0.0/24'rz:Machine ID of manager node in cluster. Must exist already.z'vastai create cluster SUBNET MANAGER_IDzCreate Vast clusterzG Create Vast Cluster by defining a local subnet and manager id.cCsb|j|jd}|jrtdt|t|d}t|||d}||jr'|St|ddS)N)rrr /cluster/rVr) rrr^r+r`rrTrrVrrrrr%r%r&create__clusterps  rAr%zEnvironment variable namerzEnvironment variable valuez$vastai create env-var z&Create a new user environment variable)rrcCpt|d}|j|jd}t||t|d}||}|dr+t|ddd Std|ddd S) z7Create a new environment variable for the current user. /secrets/rrrrrz*Environment variable created successfully.z'Failed to create environment variable: Unknown errorN) r`r%rrrPrTrVrSr+rrWrrr?r%r%r&create__env_var  rGzadd your existing ssh public key to your account (from the .pub file). If no public key is provided, a new key pair will be generated.r*)rrDr,z-yz--yesz#automatically answer yes to promptsz+vastai create ssh-key [ssh_public_key] [-y]zCreate a new ssh-keya You may use this command to add an existing public key, or create a new ssh key pair and add that public key, to your Vast account. If you provide an ssh_public_key.pub argument, that public key will be added to your Vast account. All ssh public keys should be in OpenSSH format. Example: $vastai create ssh-key 'ssh_public_key.pub' If you don't provide an ssh_public_key.pub argument, a new Ed25519 key pair will be generated. Example: $vastai create ssh-key The generated keys are saved as ~/.ssh/id_ed25519 (private) and ~/.ssh/id_ed25519.pub (public). Any existing id_ed25519 keys are backed up as .backup_. The public key will be added to your Vast account. All ssh public keys are stored in your Vast account and can be used to connect to instances they've been added to. cCsZ|j}|s t|j}ntdt|d}t||td|id}|td| dS)Nz,Adding provided SSH public key to account.../ssh/r5rz]ssh-key created {} Note: You may need to add the new public key to any pre-existing instances) r5generate_ssh_keyyesr+r`rrPrTrrV)rssh_key_contentrWrr%r%r&create__ssh_keys  rMc Cstdtd}|d}|d}z |jdddWn!ty<}ztd|tjd td WYd }~nd }~ww|sE|rtd ||rUtd d}nt d }|dvrhtdtdt t }|d|}|d|}z"|r| |td||r| |td|Wn!ty}ztd|tjd td WYd }~nd }~wwtdz tjddgdddWntytdtjd td Ynwz%dddd t|d!d"d#td$ptd%d&d'g } tj| ddd(dd)} WnStjy=}z$td*|tjd |jr.td+|jtjd td WYd }~n&d }~wty^}ztd,|tjd td WYd }~nd }~wwz|d-Wnty}ztd.|tjd WYd }~nd }~wwz!t|d/} | } Wd | WS1swY| WSty}ztd0|tjd td WYd }~d Sd }~ww)1aU Generate a new SSH key pair using ssh-keygen and return the public key content. Args: auto_yes (bool): If True, automatically answer yes to prompts Returns: str: The content of the generated public key Raises: SystemExit: If ssh-keygen is not available or key generation fails zVNo SSH key provided. Generating a new SSH key pair and adding public key to account...r id_ed25519zid_ed25519.pubiT)modeexist_okzError creating .ssh directory: rr7Nz/An SSH key pair 'id_ed25519' already exists in z/Auto-answering yes to backup existing key pair.rbz_Would you like to generate a new key pair and backup your existing id_ed25519 key pair. [y/N]: )rbrKzAborted. No new key generated.rzid_ed25519.backup_zid_ed25519.pub.backup_z#Backed up existing private key to: z"Backed up existing public key to: z$Error backing up existing SSH keys: z?Generating new SSH key pair and adding public key to account...z ssh-keygenz--helpF)r3r5zAError: ssh-keygen not found. Please install OpenSSH client tools.-ted25519z-fz-Nrcz-CUSERUSERNAMEr6z-vast.aizy )r3r4rer5zError generating SSH key: zssh-keygen error: z(Unexpected error during key generation: iz4Warning: Could not set permissions for private key: rz$Error reading generated public key: )r+rrmkdirOSErrorr,r-rgrrerfr"rrenamer9r:r9r]rgetenvCalledProcessErrorr>chmodrrr<IOError) auto_yesssh_dirprivate_key_pathpublic_key_pathr<rXrbackup_private_pathbackup_public_pathrr?rpublic_key_contentr%r%r&rJs           rJz--template_hashztemplate hash (required, but **Note**: if you use this field, you can skip search_params, as they are automatically inferred from the template)z --template_idztemplate id (optional)-n --no-default'Disable default search param query argsz --launch_argszlaunch args string for create instance ex: "--onstart onstart_wget.sh --env '-e ONSTART_PATH=https://s3.amazonaws.com/vast.ai/onstart_OOBA.sh' --image atinoda/text-generation-webui:default-nightly --disk 64"z--endpoint_namezYdeployment endpoint name (allows multiple workergroups to share same deployment endpoint)z --endpoint_idzWdeployment endpoint id (allows multiple workergroups to share same deployment endpoint)z--test_workerszinumber of workers to create to get an performance estimate for while initializing workergroup (default 3)rz --gpu_ramz5estimated GPU RAM req (independent of search string)--search_paramszsearch param string for search offers ex: "gpu_ram>=23 num_gpus=2 gpu_name=RTX_4090 inet_down>200 direct_port_count>2 disk_space>=64"z --min_loadzw[NOTE: this field isn't currently used at the workergroup level] minimum floor load in perf units/s (token/s for LLms)z --target_utilz}[NOTE: this field isn't currently used at the workergroup level] target capacity utilization (fraction, max 1.0, default 0.9)z --cold_multz[NOTE: this field isn't currently used at the workergroup level]cold/stopped instance capacity target as multiple of hot capacity target (default 2.0)z--cold_workersz9min number of workers to keep 'cold' for this workergroupz--auto_instanceunusedprodz#vastai workergroup create [OPTIONS]zCreate a new autoscale groupz Create a new autoscaling group to manage a pool of worker instances. Example: vastai create workergroup --template_hash HASH --endpoint_name "LLama" --test_workers 5 cCst|d}|jr d}nd}|jdur|jnd|}d|j|j|j|j|j|j |j ||j |j |j |j|jd}|jrCtdt|t||t|d}|d|jd dvr~z td |WdStjjy}td t|t|jYdSwtd |jd t|jdS) N /autojobs/rc) verified=True rentable=True rented=Falser)rmin_load target_util cold_mult cold_workers test_workers template_hash template_id search_params launch_argsrA endpoint_name endpoint_idautoscaler_instancerrrN Content-Typezworkergroup create {}The response is not valid JSON.'The response is not JSON. Content-Type:)r` no_defaultrrr<rkrlrmrnrorprqrsrArtru auto_instancer^r+rrPrTrSrrVrRr:JSONDecodeErrorr4)rrWrrrrrr%r%r&create__workergroup:s* :r}z6minimum floor load in perf units/s (token/s for LLms)z.zwError: env variable PORTAL_CONFIG must contain at least one non-jupyter related config string if runtype is not jupyter)r)rr)r portal_configfiltered_configr%r%r&validate_portal_configs rz;id of instance type to launch (returned from search offers)z"Create instance from template infoz--userziUser to use with docker create. This breaks some images, so only use this if you are certain you need it.z--diskz"size of local disk partition in GB --image docker container image to launch--loginzIdocker login arguments for private repo authentication, surround with '' z--labelzlabel to set on the instancez --onstartz!filename to use as onstart script --onstart-cmd-contents of onstart script as single argumentz --entrypointz,override entrypoint for args launch instance--sshLaunch as an ssh instance type --jupyter7Launch as a jupyter instance instead of an ssh instance--direct1Use (faster) direct connections for jupyter & ssh --jupyter-dirlFor runtype 'jupyter', directory in instance to use to launch jupyter. Defaults to image's working directory --jupyter-lab7For runtype 'jupyter', Launch instance with jupyter labz --lang-utf8zzWorkaround for images with locale problems: install and generate locales before instance launch, and set locale to C.UTF-8z --python-utf8zJWorkaround for images with locale problems: set python's locale to C.UTF-8z--extrar---envz9env variables and port mapping options, surround with '' z--argszrlist of arguments passed to container ENTRYPOINT. Onstart is recommended for this purpose. (must be last argument))r,rz--forcez:Skip sanity checks when creating from an existing instancez--cancel-unavailzJReturn error if scheduling fails (rather than creating a stopped instance)z --bid_pricezP(OPTIONAL) create an INTERRUPTIBLE instance with per machine bid price in $/hourz--create-volume VOLUME_ASK_IDzpCreate a new local volume using an ID returned from the "search volumes" command and link it to the new instance)rrrDz --link-volumeEXISTING_VOLUME_IDzkID of an existing rented volume to link to the instance during creation. (returned from "show volumes" cmd)z --volume-sizezSSize of the volume to create in GB. Only usable with --create-volume (default 15GB)z --mount-pathzPThe path to the volume from within the new instance container. e.g. /root/volumez--volume-labelzJ(optional) A name to give the new volume. Only usable with --create-volumez0vastai create instance ID [OPTIONS] [--args ...]zCreate a new instancea Performs the same action as pressing the "RENT" button on the website at https://console.vast.ai/create/ Creates an instance from an offer ID (which is returned from "search offers"). Each offer ID can only be used to create one instance. Besides the offer ID, you must pass in an '--image' argument as a minimum. If you use args/entrypoint launch mode, we create a container from your image as is, without attempting to inject ssh and or jupyter. If you use the args launch mode, you can override the entrypoint with --entrypoint, and pass arguments to the entrypoint with --args. If you use --args, that must be the last argument, as any following tokens are consumed into the args string. For ssh/jupyter launch types, use --onstart-cmd to pass in startup script, instead of --entrypoint and --args. Examples: # create an on-demand instance with the PyTorch (cuDNN Devel) template and 64GB of disk vastai create instance 384826 --template_hash 661d064bbda1f2a133816b6d55da07c3 --disk 64 # create an on-demand instance with the pytorch/pytorch image, 40GB of disk, open 8081 udp, direct ssh, set hostname to billybob, and a small onstart script vastai create instance 6995713 --image pytorch/pytorch --disk 40 --env '-p 8081:8081/udp -h billybob' --ssh --direct --onstart-cmd "env | grep _ >> /etc/environment; echo 'starting up'"; # create an on-demand instance with the bobsrepo/pytorch:latest image, 20GB of disk, open 22, 8080, jupyter ssh, and set some env variables vastai create instance 384827 --image bobsrepo/pytorch:latest --login '-u bob -p 9d8df!fd89ufZ docker.io' --jupyter --direct --env '-e TZ=PDT -e XNAME=XX4 -p 22:22 -p 8080:8080' --disk 20 # create an on-demand instance with the pytorch/pytorch image, 40GB of disk, override the entrypoint to bash and pass bash a simple command to keep the instance running. (args launch without ssh/jupyter) vastai create instance 5801802 --image pytorch/pytorch --disk 40 --onstart-cmd 'bash' --args -c 'echo hello; sleep infinity;' # create an interruptible (spot) instance with the PyTorch (cuDNN Devel) template, 64GB of disk, and a bid price of $0.10/hr vastai create instance 384826 --template_hash 661d064bbda1f2a133816b6d55da07c3 --disk 64 --bid_price 0.1 Return value: Returns a json reporting the instance ID of the newly created instance: {'success': True, 'new_contract': 7835610} cCs|jrt|jd }||_Wdn1swY|jdur'|j|_d}iddd|jdt|jd|jd|j d |j d |j d |jd |j d |j d|jd|jd|jd|jd|jd|jd|j}|jsv|jr~t|}||d<|jdurt|}|dkrdS||d<|jdkr|j|d<d|dvrt|t|dj|jd}|jrtdt|t ||t!|d}|"|j#r|Std|$dS)zPerforms the same action as pressing the "RENT" button on the website at https://console.vast.ai/create/. :param argparse.Namespace args: Namespace with many fields relevant to the endpoint. rNrrimagerrrrextraonstart image_login python_utf8 lang_utf8use_jupyter_labrforcecancel_unavailtemplate_hash_idr6rr7rrrz /asks/{id}/rrr Started. {})%rrr onstart_cmd entrypointr parse_envr bid_pricerrrloginrrrrrrrpr6rrrrrrr`rrr^r+rrPrTrrV)rreaderrrrrWrr%r%r&create__instances~G             rz--emailzemail address to use for loginz --usernamezusername to use for loginz --passwordzpassword to use for loginz--typez host/clientzZvastai create subaccount --email EMAIL --username USERNAME --password PASSWORD --type TYPEzCreate a subaccounta@ Creates a new account that is considered a child of your current account as defined via the API key. vastai create subaccount --email bob@gmail.com --username bob --password password --type host vastai create subaccount --email vast@gmail.com --username vast --password password --type host cCsd}|jr |jdk}|j|j|j|dd}t|ddr'tdt|dSt|d}t||t |d }| |j d krG| }t|dSt|j td |j dS) ziCreates a new account that is considered a child of your current account as defined via the API key. Fhostr)r'r8r host_only parent_idr^zRequest JSON would be: N/users/rrQzFailed with error )rDrfr'r8rgetattrr+r`rrPrTrUrVr4)rrrrWrrr%r%r&create__subaccountR s*     rz --team_namezname of the teamz(vastai create-team --team_name TEAM_NAMEzCreate a new teama As of right now creating a team account will convert your current account into a team account. Once you convert your user account into a team account, this change is permanent and cannot be reversed. The created team account will inherit all aspects of your existing user account, including billing information, cloud services, and any other account settings. The user who initiates the team creation becomes the team owner. Carefully evaluate the decision to convert your user account into a team account, as this change is permanent. For more information see: https://vast.ai/docs/team/introduction cCs8t|d}t||td|jid}|t|dS)N/team/ team_namer)r`rrPrrTr+rVrrWrr%r%r& create__team s rzname of the rolez --permissionszMfile path for json encoded permissions, look in the docs for more informationz=vastai create team-role --name NAME --permissions PERMISSIONSzAdd a new role to your teamz Creating a new team role involves understanding how permissions must be sent via json format. You can find more information about permissions here: https://vast.ai/docs/cli/roles-and-permissions cCsDt|d}t|j}t||t|j|dd}|t|dS)Nz /team/roles/r%r6r) r`rr6rrPr%rTr+rVrrWr6rr%r%r&create__team_role s rcCstddtdtddtdtddtdtdd tdtd d tdtd d tdtddtdtddddtddddtddddtddtdtddddtddtdtddtdtd d!dd"d#td$d%tdtd&d'tdtd(d)ddtd*d+tdtd,d-ddgS).Nr4name of the templaterrrz --image_tagz=23 num_gpus=1 gpu_name=RTX_3090 inet_down>128 direct_port_count>3 disk_space>=192 driver_version>=535086005 rented=False" --disk_space 8.0 --ssh --direct c Cst|d}|jo |j}|jo|j}|jp|j}|jrdn|jr!dnd}|jr1|jd}|d}nd}i}|jsGddidd iddidd id }t|j|t t t } id |j d |j d |jd|jd|jd|jd|jd|d|d|jd|d|d|jd|d| d|jd|j|j |j|j d} |jrtdt| t||t| d} | z| } | drtd | d!WdSt| d"WdSt j!j"ytd#YdSw)$N /template/rr2rr"rr`TFrrCrEr@r%rr@hrefrrr jup_direct ssh_directrruse_sshrdocker_login_repo extra_filtersrecommended_disk_spacereadme)readme_visibledescprivaterrrzNew Template: templaterrx)#r`rrr2rr)rzrrr offers_fields offers_alias offers_multr%r image_tagrrrrrrrr hide_readmerpublicr^r+rrPrTrVrRr:r|) rrWrrrrrrdefault_search_queryrrrrr%r%r&create__template s           rzid of volume offerz-size in GB of volume. Default %(default)s GB.r)rr+rDzOptional name of volume.z!vastai create volume ID [options]zCreate a new volumez Creates a volume from an offer ID (which is returned from "search volumes"). Each offer ID can be used to create multiple volumes, provided the size of all volumes does not exceed the size of the offer. cC~t|jt|jd}|jr|j|d<t|d}|jr#tdt|t||t|d}| |j r4|Std | dS)Nrrr% /volumes/rrr r"rrr%r`r^r+rrPrTrrrVrr%r%r&create__volume   rzid of network volume offerz5size in GB of network volume. Default %(default)s GB.z Optional name of network volume.z)vastai create network volume ID [options]zCreate a new network volumez Creates a network volume from an offer ID (which is returned from "search network volumes"). Each offer ID can be used to create multiple volumes, provided the size of all volumes does not exceed the size of the offer. cCr)Nrr%/network_volumes/rrrrrr%r%r&create__network_volume+ rrrz)ID of cluster to create overlay on top ofzoverlay network namez-vastai create overlay CLUSTER_ID OVERLAY_NAMEz4Creates overlay network on top of a physical clusterza Creates an overlay network to allow local networking between instances on a physical clustercC\|j|jd}|jrtd|t|d}t|||d}||jr$|St|ddS)N)rr% request json: /overlay/r?r) rr%r^r+r`rrTrrVr@r%r%r&create__overlayL    rzid of apikey to removezvastai delete api-key IDzRemove an api-keycC:t|dj|jd}t||td}|t|dSNz/auth/apikeys/{id}/rrOr`rrrrPrTr+rVrr%r%r&delete__api_keyf rzid ssh key to deletezvastai delete ssh-key IDzRemove an ssh-keycCr)N /ssh/{id}/rrOrrr%r%r&delete__ssh_keyq rrzid of scheduled job to removezvastai delete scheduled-job IDzDelete a scheduled jobcCr)Nz/commands/schedule_job/{id}/rrOrrr%r%r&delete__scheduled_job| rrzID of cluster to deletez vastai delete cluster CLUSTER_IDzDelete Clusterz Delete Vast ClustercCsXd|ji}|jr td|t|d}t|||d}||jr"|St|ddS)Nrrr>r?r)rr^r+r`rrTrrVr@r%r%r&delete__cluster s   rzid of group to deletezvastai delete workergroup ID zDelete a workergroup groupz Note that deleting a workergroup doesn't automatically destroy all the instances that are associated with your workergroup. Example: vastai delete workergroup 4242 cC|j}t|d|d}d|jd}|jrtdt|t||t|d}|d|jdd vrXz td | WdSt j j yWtd t|t|j YdSwtd |jdt|j dS) Nrirr)r autojob_idrrrNrwrczworkergroup delete {}rxryrr`r^r+rrPrTrSrrVrRr:r|r4rrrWrrr%r%r&delete__workergroup $  rzid of endpoint group to deletezvastai delete endpoint ID zDelete an endpoint groupz Note that deleting an endpoint group doesn't automatically destroy all the instances that are associated with your endpoint group, nor all the workergroups. Example: vastai delete endpoint 4242 cCr) Nrrr)r endptjob_idrrrNrwrczdelete endpoint {}rxryrrr%r%r&delete__endpoint rrz#Environment variable name to deletezvastai delete env-var z"Delete a user environment variablecCslt|d}d|ji}t||t|d}||}|dr)t|ddd Std|ddd S) z4Delete an environment variable for the current user.rCrrrrz*Environment variable deleted successfully.z'Failed to delete environment variable: rEN)r`r%rrPrTrVrSr+rFr%r%r&delete__env_var s   roverlay_identifierz+ID (int) or name (str) of overlay to delete)rr,z(vastai delete overlay OVERLAY_IDENTIFIERz;Deletes overlay and removes all of its associated instancesc Cs|j}z t|}d|i}Wnttfyd|i}Ynw|jr%td|t|d}t|||d}||j r:|St| ddS)Nr overlay_namerrr?r) rr"r TypeErrorr^r+r`rrTrrV)r identifierrrrrr%r%r&delete__overlay s"  r z --template-idz!Template ID of Template to Deletez --hash-idzHash ID of Template to DeletezAvastai delete template [--template-id | --hash-id ]zDelete a Templatea Note: Deleting a template only removes the user's replationship to a template. It does not get destroyed Example: vastai delete template --template-id 12345 Example: vastai delete template --hash-id 49c538d097ad6437413b83711c9f61e8 cCst|d}|jrd|ji}n|jrd|ji}ntddS|jr0tdt|t|t|t||t|d}t|d|jdd vrjz t|d WdSt j j yitd t|t|j YdSwtd |jdt|j dS) Nrhash_idrqzFERROR: Must Specify either Template ID or Hash ID to delete a templaterrrNrwrcrrxry) r`r rqr+r^rrPrSrVrRr:r|r4rr%r%r&delete__template s0   rzid of volume contractzvastai delete volume IDzDelete a volumez Deletes volume with the given ID. All instances using the volume must be destroyed before the volume can be deleted. cCsJt|dd|jid}t||td}||jr|Std|dS)NrrrZrOz Deleted. {}) r`rrrPrTrr+rrVrr%r%r&delete__volume5 s  rcCst|dj|d}t||tid}||jr|S|jdkr<|}|dr4tdjd it dSt|ddSt|j tdjd it dS) NrrrrQrzdestroying instance {id}.rrr%) r`rrrPrTrrUrVr+rr4)rrrWrrr%r%r&destroy_instanceG s  rzid of instance to deletez;vastai destroy instance id [-h] [--api-key API_KEY] [--raw]z0Destroy an instance (irreversible, deletes data)z Perfoms the same action as pressing the "DESTROY" button on the website at https://console.vast.ai/instances/ Example: vastai destroy instance 4242 cCt|j|dS)zPerfoms the same action as pressing the "DESTROY" button on the website at https://console.vast.ai/instances/. :param argparse.Namespace args: should supply all the command-line options N)rrrr%r%r&destroy__instanceX sridszids of instance to destroy+z%vastai destroy instances [--raw] z8Destroy a list of instances (irreversible, deletes data)cCs|jD]}t||qdS) N)rr)rrr%r%r&destroy__instancesh s  rzvastai destroy teamzDestroy your teamcCs0t|d}t||td}|t|dS)NrrO)r`rrPrTr+rVrr%r%r& destroy__teams s r ssh_key_idz'id of the key to detach to the instancez$vastai detach instance_id ssh_key_idz"Detach an ssh key from an instancez0 Example: vastai detach 99999 12345 cCs>t|dj|j|jd}t||td}|t|dS)Nz!/instances/{id}/ssh/{ssh_key_id}/)rrrO) r`rrrrrPrTr+rVrr%r%r& detach__ssh} s rzid of instance to execute onCOMMANDz(bash command surrounded by single quoteszvastai execute id COMMANDz3Execute a (constrained) remote command on a machinea" Examples: vastai execute 99999 'ls -l -o -r' vastai execute 99999 'rm -r home/delete_this.txt' vastai execute 99999 'du -d2 -h' available commands: ls List directory contents rm Remote files or directories du Summarize device usage for a set of files Return value: Returns the output of the command which was executed on the instance, if successful. May take a few seconds to retrieve the results. c Cs@t|dj|jd}d|ji}|jrtdt|t||t|d}||j rMt |j |j |j d}dj|jd}|j|d<t ||||d |jd d S|jd kr|}|d rtddD]$}td|d}t|}|jd kr|j|dd}t|nq_d Sd St|d St|jtdjditd S)zExecute a (constrained) remote command on a machine. :param argparse.Namespace args: should supply all the command-line options z/instances/command/{id}/rrrrexecutez/api/v0/instances/command/{id}/rrrNrQrr333333? result_urlwriteable_pathrcrr%)r`rrrr^r+rrPrTrrrrrrUrVrrr rRrSr4r4r) rrWrrrrrr filtered_textr%r%r&r s>        rz'id of endpoint group to fetch logs fromz--levelzlog detail level (0 to 3)r7z--tailrcz,vastai get endpt-logs ID [--api-key API_KEY]z3Fetch logs for a specific serverless endpoint groupz0 Example: vastai get endpt-logs 382 c C|jtkrd|_|jp dd}|j|jd}|jr|j|d<|jr,t|dt|t||t|d}| ddd d d }|j d krd}z| }Wnt yi}ztt |t|jWYd}~nd}~ww|jrr|pq|jS||j}|r|rt||dSdSdSt|jdS) Nhttps://run.vast.aiz/get_endpoint_logs/rr[tail with request json: rinfo0info1tracedebugrr7rrrQrWserver_url_defaultrr[r&r^r+rrPrTrUrVr>r]r4rrrrWrrlevelsrr<dbg_lvlr%r%r&get__endpt_logs 2       r2z-vastai get wrkgrp-logs ID [--api-key API_KEY]z7Fetch logs for a specific serverless worker group groupc Cr#) Nr$z/get_autogroup_logs/r%r&r'rr(r)r*r+r,rQr-r/r%r%r&get__wrkgrp_logs r3r4zemail of user to be invitedz--rolezrole of user to be invitedz.vastai invite member --email EMAIL --role ROLEzInvite a team membercCslt|d|j|jdd}t||td}||jdkr'td|jddSt|jtd|jdS) Nz /team/invite/)r'rolerrOrQzsuccessfully invited z to your current teamfailed with error ) r`r'r5rrPrTrUr+r4rr%r%r&invite__member! s  r7zID of cluster to add machine torzmachine id(s) to join clusterz*vastai join cluster CLUSTER_ID MACHINE_IDSzJoin Machine to Clusterz, Join's Machine to Vast Cluster cCr)N)rrrr>r?r) rrr^r+r`rrTrrVr@r%r%r& join__cluster2 s   r8z)Overlay network name to join instance to.zInstance ID to add to overlay.z,vastai join overlay OVERLAY_NAME INSTANCE_IDz#Adds instance to an overlay networkz6 Adds an instance to a compatible overlay network.cCr)N)r%rrrr?r) r%rr^r+r`rrTrrVr@r%r%r& join__overlayN rr9zid of instance to labelrz label to setz"vastai label instance