o xiع@sdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0ddl1m2Z2ddl3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:ddl;mZ>ddl>m?Z?ddl@mAZAddlBmCZCddlDmEZEddlFmGZGddlHmIZIdd lJmKZKdd!lLmMZMd"ZNd"ZOePeQZRejSTd#ZUejSTd$ZVGd%d&d&e<ZWGd'd(d(e<ZXGd)d*d*e=ZYd]d5d6ZZd^d>d?Z[d_dDdEZ\d`dJdKZ]dadOdPZ^dbdSdTZ_dcdWdXZ`ddd[d\ZadS)ez:Implementation of KubernetesRunner class for wandb launch.) annotationsN)Iterator)Any)Api) LaunchAgent)AbstractEnvironment)AbstractRegistry)AzureContainerRegistry) LocalRegistry)Status)WANDB_K8S_LABEL_AGENT"WANDB_K8S_LABEL_AUXILIARY_RESOURCEWANDB_K8S_LABEL_MONITORWANDB_K8S_RUN_IDCustomResourceLaunchKubernetesMonitor)recursive_macro_subsanitize_identifiers_for_k8syield_containers)ExponentialBackoff retry_async) get_module) EntryPoint LaunchProject) LaunchError)CODE_MOUNT_DIR LOG_PREFIXMAX_ENV_LENGTHSPROJECT_SYNCHRONOUSget_kube_context_and_api_clientmake_k8s_label_safemake_name_dns_safe) AbstractRunAbstractRunnerkubernetes_asynciozfKubernetes runner requires the kubernetes package. Please install it with `pip install wandb[launch]`.)required)client) AppsV1Api) BatchV1Api) CoreV1Api)CustomObjectsApi)NetworkingV1Api)V1Secret) ApiException WANDB_LAUNCH_CODE_PVC_MOUNT_PATHWANDB_LAUNCH_CODE_PVC_NAMEc@sleZdZdZ   d'd(ddZed)ddZd*ddZd+ddZd,dd Z d-d!d"Z d-d#d$Z d-d%d&Z dS).KubernetesSubmittedRunz)Wrapper for a launched run on Kubernetes.defaultN batch_apir*core_apir+apps_apir) network_apir-namestr namespace str | NonesecretV1Secret | Noneauxiliary_resource_label_keyreturnNonec Cs:||_||_||_||_||_||_d|_||_||_dS)aeInitialize a KubernetesSubmittedRun. Other implementations of the AbstractRun interface poll on the run when `get_status` is called, but KubernetesSubmittedRun uses Kubernetes watch streams to update the run status. One thread handles events from the job object and another thread handles events from the rank 0 pod. These threads updated the `_status` attributed of the KubernetesSubmittedRun object. When `get_status` is called, the `_status` attribute is returned. Arguments: batch_api: Kubernetes BatchV1Api object. core_api: Kubernetes CoreV1Api object. network_api: Kubernetes NetworkV1Api object. name: Name of the job. namespace: Kubernetes namespace. secret: Kubernetes secret. Returns: None. rN) r5r6r7r8r9r; _fail_countr=r?) selfr5r6r7r8r9r;r=r?rD]/home/ubuntu/.local/lib/python3.10/site-packages/wandb/sdk/launch/runner/kubernetes_runner.py__init__^s  zKubernetesSubmittedRun.__init__cC|jS)zReturn the run id.r9rCrDrDrEidzKubernetesSubmittedRun.idc szF|jjd|j|jdIdH}dd|jD}|s(td|jWdS|jj|d|jdIdH}|r=t|WStd|WdSt yd}zt t d |WYd}~dSd}~ww) Nz job-name=label_selectorr;cSg|]}|jjqSrDmetadatar9.0pirDrDrE z3KubernetesSubmittedRun.get_logs..z"Found no pods for kubernetes job: rr9r;zNo logs for kubernetes pod(s): zFailed to get pod logs: ) r6list_namespaced_podr9r;itemswandbtermwarnread_namespaced_pod_logr: Exception termerrorr)rCpods pod_nameslogserDrDrEget_logss*    zKubernetesSubmittedRun.get_logsboolcsf |IdH}ttd|jd|j|jdvrn tdIdHq|IdH|jdkS)zzWait for the run to finish. Returns: True if the run finished successfully, False otherwise. TNJob status: finishedfailed preemptedr0rg) get_statusrYtermlogrr9stateasynciosleep_delete_secretrCstatusrDrDrEwaits zKubernetesSubmittedRun.waitr cs(t|j}|dvr|IdH|S)N)stoppedrhrgri)rrjr9rorprDrDrErjs  z!KubernetesSubmittedRun.get_statusc snz|jj|j|jdIdH|IdHWdSty6}ztd|jd|jdt||d}~ww)zCancel the run.)r;r9Nz Failed to delete Kubernetes Job  in namespace : )r5delete_namespaced_jobr;r9ror/rr:rCrarDrDrEcancels zKubernetesSubmittedRun.cancelcsHtjds |jr"|jj|jjj|jjjdIdHd|_dSdSdS)NWANDB_RELEASE_NAMErV) osenvirongetr=r6delete_namespaced_secretrPr9r;rIrDrDrEros z%KubernetesSubmittedRun._delete_secretc s:|jdurdStd|j}zq|jdf|jdf|jdf|jdf|jdf|jdfg}|D]Q\}}z.t|d|}t|d |}||j|d IdH}|jD]}||j j |jd IdHqOWq-t t fy~} zt d |d | WYd} ~ q-d} ~ wwWdSty} zt d| WYd} ~ dSd} ~ ww)N=servicejobpodr= deploymentnetwork_policylist_namespaced_delete_namespaced_)r;rMrVzCould not clean up ruz-Failed to clean up some auxiliary resources: )r?r r6r5r7r8getattrr;rXrPr9AttributeErrorr/rYrZr\) rCrMresource_cleanups api_client resource_type list_method delete_method resourcesresourcerarDrDrE$_delete_auxiliary_resources_by_labelsL      "z;KubernetesSubmittedRun._delete_auxiliary_resources_by_label)r4NN)r5r*r6r+r7r)r8r-r9r:r;r<r=r>r?r<r@rAr@r:r@r<r@rcr@r r@rA) __name__ __module__ __qualname____doc__rFpropertyrJrbrrrjrxrorrDrDrDrEr3[s  *       r3c@sPeZdZdZdddZedddZd ddZd!ddZd"ddZ d#ddZ dS)$CrdSubmittedRunz-Run submitted to a CRD backend, e.g. Volcano.groupr:versionpluralr9r;r6r+ custom_apir,r@rAcCs4||_||_||_||_||_||_||_d|_dS)aCreate a run object for tracking the progress of a CRD. Arguments: group: The API group of the CRD. version: The API version of the CRD. plural: The plural name of the CRD. name: The name of the CRD instance. namespace: The namespace of the CRD instance. core_api: The Kubernetes core API client. custom_api: The Kubernetes custom object API client. Raises: LaunchError: If the CRD instance does not exist. rN)rrrr9r;r6rrB)rCrrrr9r;r6rrDrDrErFs zCrdSubmittedRun.__init__cCrG)z"Get the name of the custom object.rHrIrDrDrErJ!rKzCrdSubmittedRun.idr<c si}z-|jjd|j|jdIdH}dd|jD}|D]}|jj||jdIdH||<qWn!tyQ}ztd|jdt |WYd}~dSd}~ww|sVdSd d|D}d |S) zGet logs for custom object.z wandb/run-id=rLNcSrNrDrOrQrDrDrErT.rUz,CrdSubmittedRun.get_logs..rVzFailed to get logs for rucSs g|] \}}d|d|qS)zPod z: rD)rRpod_namelogrDrDrErT8s  ) r6rWr9r;rXr[r/rYrZr:join)rCr`r^r_rra logs_as_arrayrDrDrErb&s*  zCrdSubmittedRun.get_logsr cst|jS)zGet status of custom object.)rrjr9rIrDrDrErj;s zCrdSubmittedRun.get_statusc slz|jj|j|j|j|j|jdIdHWdSty5}ztd|jd|jdt ||d}~ww)zCancel the custom object.)rrr;rr9NzFailed to delete CRD rtru) rdelete_namespaced_custom_objectrrr;rr9r/rr:rwrDrDrErx?s"zCrdSubmittedRun.cancelrccsT |IdH}ttd|jd||jdvr!|jdkStdIdHq)z.Wait for this custom object to finish running.TNrdrerfrgr0)rjrYrkrr9rlrmrnrprDrDrErrNs  zCrdSubmittedRun.waitN)rr:rr:rr:r9r:r;r:r6r+rr,r@rArrrrr) rrrrrFrrJrbrjrxrrrDrDrDrErs !    rcseZdZdZd<fd d Zd=ddZd>ddZ d?d@d"d#ZdAd%d&ZdAd'd(Z dAd)d*Z dBd+d,Z - .  -dCdDd7d8Z dEd:d;Z ZS)FKubernetesRunnerzLaunches runs onto kubernetes.apirbackend_configdict[str, Any] environmentrregistryrr@rAcst||||_||_dS)a+Create a Kubernetes runner. Arguments: api: The API client object. backend_config: The backend configuration. environment: The environment to launch runs into. Raises: LaunchError: If the Kubernetes configuration is invalid. N)superrFrr)rCrrrr __class__rDrErF[s zKubernetesRunner.__init__ resource_argscontextr:cCsL|r |dddnd}|didp%|dp%|jdidp%|S)zGet the namespace to launch into. Arguments: resource_args: The resource args to launch. context: The k8s config context. Returns: The namespace to launch into. rr;r4rPrunner)r|r)rCrrdefault_namespacerDrDrE get_namespaceps zKubernetesRunner.get_namespacelaunch_projectr image_urir;r6r+&tuple[dict[str, Any], V1Secret | None]c sJddd}|||di}ddd}||di|d i} d d i} | | di| d ig} |d i|j|d t<d|d t<trXt|d t <|dslt d|j d|j d|d<||d<t | D]%\} } d| vr| ddt| | d<d| vrdddgiddid| d<qt|jp|}|jr|| dd<ntdd| Ds|dusJ|| dd<t||j|j|IdH}|durdd |jig| d!<t| ||j|jdu||jt|jj}d}| D]{} | d"pg}|D]i\}}|d#krb|rbts|jtrbtj d$}d%}|r,|d|7}n|d|j7}d&d'}t!t"t#j$d(d)t#j$d(d*t%d+t&|||||d,IdH}|'|d-|d.d/id0q|'||d1q|| d"<q| | d <| | d<| |d <||d<||d<t(|td|j)rt*||trt(|t t||fS)2aApply our default values, return job dict and api key secret. Arguments: resource_args (Dict[str, Any]): The resource args to launch. launch_project (LaunchProject): The launch project. builder (Optional[AbstractBuilder]): The builder. namespace (str): The namespace. core_api (CoreV1Api): The core api. Returns: Tuple[Dict[str, Any], Optional["V1Secret"]]: The resource args and api key secret. batch/v1Job) apiVersionkindrPr<) backoffLimitttlSecondsAfterFinishedspectemplate restartPolicyNever containerslabelstruer9zlaunch-- generateNamer;launchsecurityContextFdropALLtypeRuntimeDefault)allowPrivilegeEscalation capabilitiesseccompProfileimagecSsg|]}d|vqS)rrD)rRcontrDrDrErTrUz5KubernetesRunner._inject_defaults..Nregcred-imagePullSecretsenv WANDB_API_KEYryz wandb-api-keycSstd|ddS)N3Exception when ensuring Kubernetes API key secret: z . Retrying...)rYrZ)rarDrDrEhandle_exceptions z;KubernetesRunner._inject_defaults..handle_exceptionr#)seconds)minutes) initial_sleep max_sleep max_retries)backofffnon_excr6 secret_namer;api_key secretKeyRefpasswordr9keyr9 valueFromr9value)+updater| setdefaultrun_idrrr initializedr9r r" target_entitytarget_project enumerater:override_entrypointget_job_entry_point docker_imageanymaybe_create_imagepull_secretrinject_entrypoint_and_args override_argsget_env_vars_dict_apirrrrXrrrzr{rrdatetime timedeltaAPI_KEY_SECRET_MAX_RETRIESensure_api_key_secretappendadd_label_to_podsjob_base_imageapply_code_mount_configuration)rCrrrr;r6r job_metadatajob_spec pod_templatepod_specrir entry_pointr=env_varsapi_key_secretrrr release_namerrrDrDrE_inject_defaultss                     z!KubernetesRunner._inject_defaults,r#kubernetes_asyncio.client.ApiClientconfigtimeout_secondsintc s|d}|did}|r|sttddSttd|d|dt}|d kr?|||||IdHn3|d krO|||||IdHn#|d kr_|||||IdHnttd |d t dIdHt|}tt|d|d|dddS)a1Wait for a Kubernetes resource to be ready. Arguments: api_client: The Kubernetes API client. config: The resource configuration. namespace: The namespace where the resource was created. timeout_seconds: Maximum time to wait for readiness. rrPr9z-Cannot wait for resource without kind or nameNz Waiting for z 'z' to be ready... DeploymentServicePodz No specific readiness check for z, waiting 5 seconds...r0z' is ready after z.1fs) r|rYr]rrktime_wait_for_deployment_ready_wait_for_service_ready_wait_for_pod_readyrmrn) rCrr r;r resource_kind resource_name start_timeelapsedrDrDrE_wait_for_resource_ready1s@     z)KubernetesRunner._wait_for_resource_readyr9c6tj|fdd}|||IdHdS)z"Wait for a Deployment to be ready.cs8jdIdH}|j}|jr|jr|j|jkSdS)NrVF)read_namespaced_deploymentrqready_replicasreplicas)rrqr7r9r;rDrEcheck_deployment_readyps   zKKubernetesRunner._wait_for_deployment_ready..check_deployment_readyN)r&r(r)_wait_with_timeout)rCrr9r;rr"rDr!rErfs  z+KubernetesRunner._wait_for_deployment_readycr)z%Wait for a Service to have endpoints.cs8jdIdH}|jr|jD]}|jrdSqdS)NrVTF)read_namespaced_endpointssubsets addresses) endpointssubsetr6r9r;rDrEcheck_service_readys  zEKubernetesRunner._wait_for_service_ready..check_service_readyNr&r(r+r#)rCrr9r;rr*rDr)rEr}s  z(KubernetesRunner._wait_for_service_readycr)zWait for a Pod to be ready.csHjdIdH}|jjdkr"|jjr tdd|jjDSdSdS)NrVRunningcss|]}|jVqdS)N)ready)rRrqrDrDrE szPKubernetesRunner._wait_for_pod_ready..check_pod_ready..TF)read_namespaced_podrqphasecontainer_statusesall)rr)rDrEcheck_pod_readys z=KubernetesRunner._wait_for_pod_ready..check_pod_readyNr+)rCrr9r;rr3rDr)rErs z$KubernetesRunner._wait_for_pod_readyc st}t||krjz |IdHrWdSWnAtjjy@}z|jdkr(nttd|d|WYd}~nd}~wtyY}zttd|d|d}~wwt dIdHt||ks t d|d|d) z-Generic timeout wrapper for readiness checks.NizError waiting for resource 'z': rz Resource 'z' not ready within z seconds) rr&r(r/rqrYr]rr\rmrnr)rC check_funcrr9rrarDrDrEr#s4   z#KubernetesRunner._wait_with_timeoutNTrrr>wait_for_readyrc wait_timeoutauxiliary_resource_label_valuer<c  sX|di|ddi||ddt<d|ddd<| r)| |ddt<||jt|jj} d| ddi} t || | rIt |t| |rkt |D]} | dg} | dd |j jd d id | | d<qOz#t|tjj|||d IdH|r|||||IdHWdSWdSty}zttd|td|d}~ww)aZPrepare a service for launch. Arguments: api_client: The Kubernetes API client. config: The resource configuration to prepare. namespace: The namespace to create the resource in. run_id: The run ID to label the resource with. launch_project: The launch project to get environment variables from. api_key_secret: The API key secret to inject. wait_for_ready: Whether to wait for the resource to be ready after creation. wait_timeout: Maximum time in seconds to wait for resource readiness. rPr launch-agentwandb.ai/created-by WANDB_CONFIGz{}rrrrrrr;Nz&Failed to create Kubernetes resource: )rrr rrrrrr| add_wandb_envrrrrPr9rr&utilscreate_from_dictrr\rYr]rr)rCrr r;rrrr5r6r7rwandb_config_envrrrarDrDrE_prepare_resources`       z"KubernetesRunner._prepare_resourceAbstractRun | Nonec$ stIdH|di}|sttdtd|t t |IdH\}j durLt dus:t dur>td}tjt ||dd}|dvrijtjj}t|||d i|d <|d d i|d d <d |d d t<t|td trt|ttt|d d t<j rt |i}j!rj!|d <j"rj"j#|d <t$||t%&} '|||(d^} } } |d| } |d| } |d| } | )d}t*| | |d}tj+|dz| j,| | ||dIdH}Wn+t-y3}zt./|j0}t12|}td| d|j3d|j4d||d}~ww|d id}td| d|d dt5|| | |t%6| d}j7t8rg|9IdH|St j%:}t j%6}t j%;}t j%<}'||=|||IdH\}j>j?j@j|jAd}|BtjCtDjEdg|}tFdj?d j>d j@|rttd!||d"d#|d$d%tGjHfd&d'|DIdHd(}d|vr|d|d7}t|zt jIjJ|d)IdH}WnRt jIjKyQ}z-|jLD]#}t./|j0}|d*}|d+} td,j@d-| d|j4d.|WYd}~nd}~wtMyg}z td/tN|dd}~ww|d0}!|!jOj}"t+tP|||||"}#j7t8r|#9IdH|#S)1aExecute a launch project on Kubernetes. Arguments: launch_project: The launch project to execute. builder: The builder to use to build the image. Returns: The run object if the run was successful, otherwise None. N kuberneteszyNote: no resource args specified. Add a Kubernetes yaml spec or other options in a json file with --resource-args .z+Running Kubernetes job with resource args: a0WANDB_LAUNCH_SOURCE_CODE_PVC_ environment variables not set. Unable to mount source code PVC into base image. Use the `codeMountPvcName` variable in the agent helm chart to enable base image jobs for this agent. See https://github.com/wandb/helm-charts/tree/main/charts/launch-agent for more information.rr)rz batch/v1beta1rPrrargscommand/rrrr)rrr)custom_resource)rrr;rbodyzError creating CRD of kind ru rr9zCreated )r9rrr;rr6r) project_name entity_namerrun_namerauthoradditional_serviceszaux-rzCreating additional services: r5Tr6r c s<g|]}|dir|dij qS)r )r|r@r)rRrrr7rr;r=rCr5r6rDrErTs   z(KubernetesRunner.run..zCreating Kubernetes jobr;messagecodez(Failed to create Kubernetes job for run z (z): z3Unexpected exception when creating Kubernetes job: r)Qrensure_initialized fill_macrosr|rYrkr_loggerinfor r&rSOURCE_CODE_PVC_NAMESOURCE_CODE_PVC_MOUNT_PATHrget_image_source_stringchange_project_dirrzpathrrrrrrr<rrrrr r9rrrrDadd_entrypoint_args_overridesr(r,rsplitlowerrmonitor_namespacecreate_namespaced_custom_objectr/jsonloadsrGyamldumprqreasonrr+rrrrr*r)r-r rrrrLrr{r launch_specr!rmgatherr=r>FailToCreateErrorapi_exceptionsr\r:rPr3)$rCrrrr code_subdir api_versionr overridesrrr_rrrFresponserarG body_yamlr9 submitted_runr5r6r7r8r update_dictrMmsgexcresprP job_responsejob_name submitted_jobrDrNrEruns^                                       zKubernetesRunner.run) rrrrrrrrr@rA)rrrrr@r:) rrrrrr:r;r:r6r+r@r)r ) rr r rr;r:rrr@rA) rr r9r:r;r:rrr@rA)rrr9r:r@rA)NTr N)rr r rr;r:rr:rrrr>r5rcr6rr7r<r@rA)rrrr:r@rA)rrrrrFrr rrrrr#r@rv __classcell__rDrDrrErXs$  / 5   " Orr list[dict]rEntryPoint | Noner list[str]should_override_entrypointrcr@rAcCsJtt|D]}|r|||d<|r"||dr|r"|j||d<qdS)aPInject the entrypoint and args into the containers. Arguments: containers: The containers to inject the entrypoint and args into. entry_point: The entrypoint to inject. override_args: The args to inject. should_override_entrypoint: Whether to override the entrypoint. Returns: None rCrDN)rangelenr|rD)rrrr{rrDrDrErs  rr6r+rr:r;rr.c s2dt|i}ddi}tj|tj|||dddd}zcz |||IdHWWSty}zK|j d kr}|j ||d IdH}|j |krt|j j ddkrj|j||d IdH|||IdHWYd}~WStd |d ||WYd}~WSd}~wwty}z td t|dd}~ww)a<Create a secret containing a user's wandb API key. Arguments: core_api: The Kubernetes CoreV1Api object. secret_name: The name to use for the secret. namespace: The namespace to create the secret in. api_key: The user's wandb API key Returns: The created secret rr9r8)r9r;rSecretzkubernetes.io/basic-authdatarPrrNrVz.Kubernetes secret already exists in namespace z with incorrect data: rr)base64 b64encodeencodedecoder(r. V1ObjectMetacreate_namespaced_secretr/rqread_namespaced_secretrrPrr|r}rr\r:) r6rr;r secret_datarr=raexisting_secretrDrDrErsT     rrrrr>c s&d}t|ttfr dS|IdH\}}d|jt|d|ddii}dtt |i}t j |t j d||dd d d }z2z |||IdHWWSty|} z| jd krw|jd||dIdHWYd} ~ WSd} ~ wwty} z td t| dd} ~ ww)aICreate a secret for pulling images from a private registry. Arguments: core_api: The Kubernetes CoreV1Api object. registry: The registry to pull from. run_id: The run id. namespace: The namespace to create the secret in. Returns: A secret if one was created, otherwise None. Nauths:zdeprecated@wandblaunch.com)authemailz.dockerconfigjsonrrVr~zkubernetes.io/dockerconfigjsonrrz+Exception when creating Kubernetes secret: r) isinstancer r get_username_passwordurirrrrr_dumpsr(r.rrr/rqrr\rr:) r6rrr;r=unametoken creds_inforrarDrDrErMsF   rroot dict | listrdict[str, str]cCsPt|D]!}|dg}|dd|D||d<d|vr%|dqdS)aInjects wandb environment variables into specs. Recursively walks the spec and injects the environment variables into every container spec. Containers are identified by the "containers" key. This function treats the WANDB_RUN_ID and WANDB_GROUP_ID environment variables specially. If they are present in the spec, they will be overwritten. If a setting for WANDB_RUN_ID is provided in env_vars, then that environment variable will only be set in the first container modified by this function. Arguments: root: The spec to modify. env_vars: The environment variables to inject. Returns: None. rcSsg|] \}}||dqS)rrD)rRrrrDrDrErTsz!add_wandb_env.. WANDB_RUN_IDN)rrextendrXpop)rrrrrDrDrEr<s   r<manifestrIterator[dict]ccs~t|tr|D] }t|EdHqdSt|tr;d|vr&d|dvr&|V|D]}t|ttfr:t|EdHq*dSdS)zYield all pod specs in a manifest. Recursively traverses the manifest and yields all pod specs. Pod specs are identified by the presence of a "spec" key with a "containers" key in the value. Nrr)rlist yield_podsdictvalues)ritemrrDrDrErs   r label_key label_valuecCs2t|D]}|di}|di}|||<qdS)aAdd a label to all pod specs in a manifest. Recursively traverses the manifest and adds the label to all pod specs. Pod specs are identified by the presence of a "spec" key with a "containers" key in the value. Arguments: manifest: The manifest to modify. label_key: The label key to add. label_value: The label value to add. Returns: None. rPrN)rr)rrrrrPrrDrDrErs    rrjrcCst|tr|D]}t||qdSt|trKd|vr?d|dvr?|dd}|D]}d|vr4|d|d<d|vr>|d|d<q(|D] }t||qCdSdS)aAdd entrypoint and args overrides to all containers in a manifest. Recursively traverses the manifest and adds the entrypoint and args overrides to all containers. Containers are identified by the presence of a "spec" key with a "containers" key in the value. Arguments: manifest: The manifest to modify. overrides: Dictionary with args and entrypoint keys. Returns: None. rrrDrCN)rrrZrr)rrjrr containerrrDrDrErZs"        rZprojectrcCstdusJ|}t|D]8}t|D]}d|vrg|d<|ddt|dt|d<q|d}d|vr:g|d<|dddtid qdS) aApply code mount configuration to all containers in a manifest. Recursively traverses the manifest and adds the code mount configuration to all containers. Containers are identified by the presence of a "spec" key with a "containers" key in the value. Arguments: manifest: The manifest to modify. project: The launch project. Returns: None. N volumeMountszwandb-source-code-volume)r9 mountPathsubPath workingDirrvolumes claimName)r9persistentVolumeClaim)rUrWrrrr)rr source_dirrrrrDrDrErs.    r) rrxrryrrzr{rcr@rA) r6r+rr:r;r:rr:r@r.) r6r+rrrr:r;r:r@r>)rrrrr@rA)rrr@r)rrrr:rr:r@rA)rrrjrr@rA)rrrrr@rA)br __future__rrmrrr_loggingrzrcollections.abcrtypingrrarYwandb.apis.internalrwandb.sdk.launch.agent.agentr%wandb.sdk.launch.environment.abstractr"wandb.sdk.launch.registry.abstractr2wandb.sdk.launch.registry.azure_container_registryr (wandb.sdk.launch.registry.local_registryr wandb.sdk.launch.runner.abstractr *wandb.sdk.launch.runner.kubernetes_monitorr r rrrrwandb.sdk.launch.utilsrrrwandb.sdk.lib.retryrr wandb.utilr _project_specrrerrorsrr=rrrrr r!r"abstractr$r%r&r()kubernetes_asyncio.client.api.apps_v1_apir)*kubernetes_asyncio.client.api.batch_v1_apir*)kubernetes_asyncio.client.api.core_v1_apir+0kubernetes_asyncio.client.api.custom_objects_apir,/kubernetes_asyncio.client.api.networking_v1_apir-*kubernetes_asyncio.client.models.v1_secretr.kubernetes_asyncio.client.restr/TIMEOUTr getLoggerrrSr{r|rVrUr3rrrrrr<rrrZrrDrDrDrEs~             $            #[ #  = 6