o 2wi@sdZddlmZddlmZmZddlmZddlmZddl Z ddl m Z ddl mZd e jd efd d Zd e jd e fddZeGdddZeGdddZeGdddZeGdddZdS)z' Basic configuration for Dora is here. ) Namespace) dataclassfield)fnmatch)PathN) DictConfig) OmegaConfdataargscCsF|jD]}t|ts Jt||r t||}|dur t|||qdS)z>Update the given dataclass from the argument parser args. N)__dict__ isinstancestrhasattrgetattrsetattr)r r keyvaluerF/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/dora/conf.pyupdate_from_argss    rcfgcCsltj|dd}t|tsJ|D]!\}}t|tsJt||r)t|||qtd|j d|dS)z6Update the given dataclass from the hydra config. T)resolvezObject of type z does not have an attribute N) r to_containerr dictitemsr rrAttributeError __class__)r rdctrrrrrupdate_from_hydras  rc@seZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z e j eed <d Zeed <d Ze j eed<eedZe jeed<dZeed<dZeed<dZeed<dZeed<d Ze j eed<d Ze j eed<d Ze j eed<d S) SlurmConfiga+ Configuration when scheduling a job. This differs slightly from Slurm/Submitit params because this will automatically scale some values based on the number of GPUs. Args: gpus (int): number of total GPUs to schedule. Number of nodes and tasks per nodes will be automatically inferred. mem_per_gpu (float): amount of memory in GB to schedule per gpus. time (int): maximum duration for the job in minutes. cpus_per_gpu (int): number of cpus per gpu, this will set the `cpus_per_task` automatically, based on the number of gpus and `one_task_per_node`, unless `cpus_per_task` is explicitely provided. cpus_per_task (int or None): number of cpus per task. partition (str): partition name comment (str): comment for the job. setup (List[str]): list of shell commands to execute before the actual command. Use it for `module load`. max_num_timeout (int): maximum number of requeue. one_task_per_node (bool): if True, schedules a single task per node, otherwise, will schedule one task per gpu (default is False). array_parallelism (int): when using job arrays, how many tasks can run in parallel. qos: (str or None): qos param for slurm. account: (str or None): account param for slurm. ..warning:: this assumes one task per GPU. Set `one_task_per_node` if you do not want that. Tasks without any gpus are not really supported at the moment. gpus( mem_per_gpuitime cpus_per_gpuN cpus_per_tasklearnlab partitioncommentdefault_factorysetupmax_num_timeout constraintFone_task_per_nodearray_parallelismexcludeqosaccount)__name__ __module__ __qualname____doc__r!int__annotations__r#floatr$r&r'tpOptionalr)r r*rlistr-Listr/r1r2boolr4r5r6r7rrrrr.s"         rc@sBeZdZUdZdZeed<dZeed<dZeed<dZ eed<dS) SubmitRulesa: Submit rules describe in which case Shepherd will schedule new jobs. Args: retry (bool): if true, all failed or canceled jobs will be rescheduled. update_pending (bool): if true, all pending jobs whose Slurm parameters have changed will be replaced. update (bool): if true, all pending or running jobs whose Slurm parameters have changed will be replaced. replace (bool): if true, all running jobs will be replaced by new jobs. replace_done (bool): if true, all done jobs will be relaunched. Fretryupdatereplace replace_doneN) r8r9r:r;rErCr=rFrGrHrrrrrDas   rDc@sZeZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed <d Z eed <d S) ShepConfigz Configuration for Shepherd. Mostly naming conventions for folders and files. There should be little reasons to change that. zjob.pkljob_fileby_idorphanssubmititsubmitit_folderlatestlatest_submititarraysN) r8r9r:r;rJr r=rKrLrNrPrQrrrrrIvs      rIcseZdZUdZedZeed<eedZ e j e ed<dZ eed<dZe jeed <dZe je ed <d Ze ed <d Ze ed <eedZeed<dZe ed<dZeed<dZe ed<dZe ed<de defddZfddZZS) DoraConfiga< Main Dora configuration. The main parameters to change are the following. Args: dir (Path or str): path where Dora will save all useful informations, logs. This is also where you should store your checkpoints (see `dora.xp.XP`). exclude (List[str]): list of patterns of argument names to ignore when computing the XP signature and doing deduplication. For instance 'num_workers', etc. git_save (bool): when True, experiments can only be scheduled from a clean repo. A shallow clone of the repo will be made and execution will happen from there. This does not impact `dora run` unless you pass the `--git_save` flag. shared (Path or None): if provided, the path to a central repository of XPs. For the moment, this only supports sharing hyper-params, logs etc. will stay in the per user folder. grid_package (str or None): if provided, package to look for grids. Default to the package with the `train.py` module followed by `.grids`. z ./outputsdirr+r5Fgit_saveNshared grid_packagez history.jsonhistoryxpsshepzrendezvous.txtrendezvous_fileuse_rendezvousgrids_gridscodes_codesarg_namereturncCs |jD] }t||r dSqdS)zUReturn True if the given argument name should be excluded from the signature.TF)r5r)selfr`patternrrr is_excludeds  zDoraConfig.is_excludedcs:|dvrddlm}|durt||}t||dS)N)rSrUr )to_absolute_path)rTrersuper __setattr__)rbnamerrerrrrgs   zDoraConfig.__setattr__)r8r9r:r;rrSr=rrAr5r?rBr rTrCrUr@rVrWrXrIrYrZr[r]r_rdrg __classcell__rrrirrRs        rR)r;argparser dataclassesrrrpathlibrtypingr?omegaconf.dictconfigr omegaconfrAnyrrrrDrIrRrrrrs$      2