o oipa@s(dZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z mZddlmZddlmZddlmZddlmZdd lmZmZmZmZdd lmZdd lmZdd l m!Z!m"Z"dd l#m$Z$ddl%m&Z&ddl'm(Z(m)Z)erddl*m+Z+ddl,m-Z-ddl.m/Z/edZ0Gddde!Z1dS)z5 Weights and Biases Logger ------------------------- N) Namespace)Path) TYPE_CHECKINGAnyDictListLiteralMappingOptionalUnion)RequirementCache)Tensor)override) _add_prefix_convert_json_serializable_convert_params_sanitize_callable_params)_PATH)ModelCheckpoint)Loggerrank_zero_experiment)_scan_checkpoints)MisconfigurationException)rank_zero_onlyrank_zero_warn)Artifact RunDisabledRunzwandb>=0.12.10csLeZdZdZdZ            dUdeeded eed ed eed eed eedeede e defde ddedeede ddffdd Z de ee ffddZeede dfddZ dVd!ejd"eed#ed$eddf d%d&Zeed'e e ee fefddfd(d)ZeedWd*eeefd+eeddfd,d-Ze    dXd.ed/eeed0eeee d1e d+eeddf d2d3Ze    dXd.ed/eeed0eeeed1e d+eeddf d4d5ZedWd.ed6ee d+eede ddf d7d8Z edWd.ed9ee d+eede ddf d:d;Z!edWd.edZ"eedeefd?d@Z#eedeefdAdBZ$eedeefdCdDZ%edEe&ddfdFdGZ'e(e   dYdHedeedIeedJeedef dKdLZ)dWdHedIeeddMfdNdOZ*eedPeddfdQdRZ+dEe&ddfdSdTZ,Z-S)Z WandbLoggera Log using `Weights and Biases `_. **Installation and set-up** Install with pip: .. code-block:: bash pip install wandb Create a `WandbLogger` instance: .. code-block:: python from lightning.pytorch.loggers import WandbLogger wandb_logger = WandbLogger(project="MNIST") Pass the logger instance to the `Trainer`: .. code-block:: python trainer = Trainer(logger=wandb_logger) A new W&B run will be created when training starts if you have not created one manually before with `wandb.init()`. **Log metrics** Log from :class:`~lightning.pytorch.core.LightningModule`: .. code-block:: python class LitModule(LightningModule): def training_step(self, batch, batch_idx): self.log("train/loss", loss) Use directly wandb module: .. code-block:: python wandb.log({"train/loss": loss}) **Log hyper-parameters** Save :class:`~lightning.pytorch.core.LightningModule` parameters: .. code-block:: python class LitModule(LightningModule): def __init__(self, *args, **kwarg): self.save_hyperparameters() Add other config parameters: .. code-block:: python # add one parameter wandb_logger.experiment.config["key"] = value # add multiple parameters wandb_logger.experiment.config.update({key1: val1, key2: val2}) # use directly wandb module wandb.config["key"] = value wandb.config.update() **Log gradients, parameters and model topology** Call the `watch` method for automatically tracking gradients: .. code-block:: python # log gradients and model topology wandb_logger.watch(model) # log gradients, parameter histogram and model topology wandb_logger.watch(model, log="all") # change log frequency of gradients and parameters (100 steps by default) wandb_logger.watch(model, log_freq=500) # do not log graph (in case of errors) wandb_logger.watch(model, log_graph=False) The `watch` method adds hooks to the model which can be removed at the end of training: .. code-block:: python wandb_logger.experiment.unwatch(model) **Log model checkpoints** Log model checkpoints at the end of training: .. code-block:: python wandb_logger = WandbLogger(log_model=True) Log model checkpoints as they get created during training: .. code-block:: python wandb_logger = WandbLogger(log_model="all") Custom checkpointing can be set up through :class:`~lightning.pytorch.callbacks.ModelCheckpoint`: .. code-block:: python # log model only if `val_accuracy` increases wandb_logger = WandbLogger(log_model="all") checkpoint_callback = ModelCheckpoint(monitor="val_accuracy", mode="max") trainer = Trainer(logger=wandb_logger, callbacks=[checkpoint_callback]) `latest` and `best` aliases are automatically set to easily retrieve a model checkpoint: .. code-block:: python # reference can be retrieved in artifacts panel # "VERSION" can be a version (ex: "v2") or an alias ("latest or "best") checkpoint_reference = "USER/PROJECT/MODEL-RUN_ID:VERSION" # download checkpoint locally (if not already cached) run = wandb.init(project="MNIST") artifact = run.use_artifact(checkpoint_reference, type="model") artifact_dir = artifact.download() # load checkpoint model = LitModule.load_from_checkpoint(Path(artifact_dir) / "model.ckpt") **Log media** Log text with: .. code-block:: python # using columns and data columns = ["input", "label", "prediction"] data = [["cheese", "english", "english"], ["fromage", "french", "spanish"]] wandb_logger.log_text(key="samples", columns=columns, data=data) # using a pandas DataFrame wandb_logger.log_text(key="samples", dataframe=my_dataframe) Log images with: .. code-block:: python # using tensors, numpy arrays or PIL images wandb_logger.log_image(key="samples", images=[img1, img2]) # adding captions wandb_logger.log_image(key="samples", images=[img1, img2], caption=["tree", "person"]) # using file path wandb_logger.log_image(key="samples", images=["img_1.jpg", "img_2.jpg"]) More arguments can be passed for logging segmentation masks and bounding boxes. Refer to `Image Overlays documentation `_. **Log Tables** `W&B Tables `_ can be used to log, query and analyze tabular data. They support any type of media (text, image, video, audio, molecule, html, etc) and are great for storing, understanding and sharing any form of data, from datasets to model predictions. .. code-block:: python columns = ["caption", "image", "sound"] data = [["cheese", wandb.Image(img_1), wandb.Audio(snd_1)], ["wine", wandb.Image(img_2), wandb.Audio(snd_2)]] wandb_logger.log_table(key="samples", columns=columns, data=data) **Downloading and Using Artifacts** To download an artifact without starting a run, call the ``download_artifact`` function on the class: .. code-block:: python from lightning.pytorch.loggers import WandbLogger artifact_dir = WandbLogger.download_artifact(artifact="path/to/artifact") To download an artifact and link it to an ongoing run call the ``download_artifact`` function on the logger instance: .. code-block:: python class MyModule(LightningModule): def any_lightning_module_function_or_hook(self): self.logger.download_artifact(artifact="path/to/artifact") To link an artifact from a previous run you can use ``use_artifact`` function: .. code-block:: python from lightning.pytorch.loggers import WandbLogger wandb_logger = WandbLogger(project="my_project", name="my_run") wandb_logger.use_artifact(artifact="path/to/artifact") See Also: - `Demo in Google Colab `__ with hyperparameter search and model logging - `W&B Documentation `__ Args: name: Display name for the run. save_dir: Path where data is saved. version: Sets the version, mainly used to resume a previous run. offline: Run offline (data can be streamed later to wandb servers). dir: Same as save_dir. id: Same as version. anonymous: Enables or explicitly disables anonymous logging. project: The name of the project to which this run will belong. If not set, the environment variable `WANDB_PROJECT` will be used as a fallback. If both are not set, it defaults to ``'lightning_logs'``. log_model: Log checkpoints created by :class:`~lightning.pytorch.callbacks.ModelCheckpoint` as W&B artifacts. `latest` and `best` aliases are automatically set. * if ``log_model == 'all'``, checkpoints are logged during training. * if ``log_model == True``, checkpoints are logged at the end of training, except when :paramref:`~lightning.pytorch.callbacks.ModelCheckpoint.save_top_k` ``== -1`` which also logs every checkpoint during training. * if ``log_model == False`` (default), no checkpoint is logged. prefix: A string to put at the beginning of metric keys. experiment: WandB experiment object. Automatically set when creating a run. checkpoint_name: Name of the model checkpoint artifact being logged. \**kwargs: Arguments passed to :func:`wandb.init` like `entity`, `group`, `tags`, etc. Raises: ModuleNotFoundError: If required WandB package is not installed on the device. MisconfigurationException: If both ``log_model`` and ``offline`` is set to ``True``. -N.Fnamesave_dirversionofflinedirid anonymousproject log_modelall experiment)rrNprefixcheckpoint_namekwargsreturnc  s tsttt|r| rtd| d|dt||_| |_| |_| |_ i|_ d|_ |dur8t |}n |durAt |}|pIt jdd}|||pO||pR|d|rWdndd|_|jjd i| |jd|_|jd |_|jd |_|jd |_| |_dS) NzProviding log_model=z and offline=z is an invalid configuration since model checkpoints cannot be uploaded in offline mode. Hint: Set `offline=False` to log your model. WANDB_PROJECTlightning_logsallow)r$r+r(r)resumer*r+r(r$r))_WANDB_AVAILABLEModuleNotFoundErrorstrrsuper__init___offline _log_model_prefix _experiment_logged_model_time_checkpoint_callbackosfspathenvironget _wandb_initupdate_project _save_dir_name_id_checkpoint_name)selfr$r%r&r'r(r)r*r+r,r.r/r0r1 __class__r7S/home/ubuntu/.local/lib/python3.10/site-packages/lightning/pytorch/loggers/wandb.pyr<$s>      zWandbLogger.__init__cCshddl}|d|j}|j}|jdur.t|jdd|d<t|jdd|d<|jj|d<d|d<|S)Nrservicer)rL _attach_idrKr@)wandbrequirer.__dict__copyr@getattrr$)rNrT_stater7r7rQ __getstate___s    zWandbLogger.__getstate__)rrcCsddl}ddlm}ddlm}|jdurm|jrdtjd<t |dd}|j dur3t d|j |_|jS|durEt |d rE| ||_|jS|jdi|j|_t|j||frmt |jd drm|jd |jjd d d d|jS)zActual wandb object. To use wandb features in your :class:`~lightning.pytorch.core.LightningModule` do the following. Example:: .. code-block:: python self.logger.experiment.some_wandb_function() rNrrdryrun WANDB_MODErSzThere is a wandb run already in progress and newly created instances of `WandbLogger` will reuse this run. If this is not desired, call `wandb.finish()` before instantiating `WandbLogger`._attach define_metrictrainer/global_step*T) step_metric step_syncr7)rT wandb.sdk.librwandb.wandb_runrr@r=rCrErXrunrhasattrr^initrG isinstancer_)rNrTrr attach_idr7r7rQr.ts.         zWandbLogger.experiment gradientsdTmodelloglog_freq log_graphcCs|jj||||ddS)N)rnrorp)r.watch)rNrmrnrorpr7r7rQrqszWandbLogger.watchparamscCs.t|}t|}t|}|jjj|dddS)NT)allow_val_change)rrrr.configrH)rNrrr7r7rQlog_hyperparamsszWandbLogger.log_hyperparamsmetricsstepcCsZtjdks Jdt||j|j}|dur%|jt|fid|idS|j|dS)Nrz-experiment tried to log from global_rank != 0r`)rrankrr?LOGGER_JOIN_CHARr.rndict)rNrvrwr7r7rQ log_metricss  zWandbLogger.log_metricskeycolumnsdata dataframecCs,ddl}||j|||di}|||dS)zLog a Table containing any object type (text, image, audio, video, molecule, html, etc). Can be defined either with `columns` and `data` or with `dataframe`. rN)r}r~r)rTTabler{)rNr|r}r~rrwrTrvr7r7rQ log_tableszWandbLogger.log_tablecCs||||||dS)zlLog text as a Table. Can be defined either with `columns` and `data` or with `dataframe`. N)r)rNr|r}r~rrwr7r7rQlog_textszWandbLogger.log_textimagesc  t|tstdt|t|}D]\}}t||kr/td|dt|d|qfddt|D}ddl|fd dt ||Di} | | |dS) zLog images (tensors, numpy arrays, PIL Images or file paths). Optional kwargs are lists passed to each image (ex: caption, masks, boxes). z#Expected a list as "images", found Expected  items but only found  for c g|] fddDqS)ci|] }||qSr7r7.0kir1r7rQ z4WandbLogger.log_image...r7rr1rrQ  z)WandbLogger.log_image..rNc"g|] \}}j|fi|qSr7)Image)rimgkwargrTr7rQr" rilist TypeErrortypelenitems ValueErrorrangerTzipr{) rNr|rrwr1nrv kwarg_listrvr7r1rTrQ log_images  zWandbLogger.log_imageaudiosc  r) aLog audios (numpy arrays, or file paths). Args: key: The key to be used for logging the audio files audios: The list of audio file paths, or numpy arrays to be logged step: The step number to be used for logging the audio files \**kwargs: Optional kwargs are lists passed to each ``Wandb.Audio`` instance (ex: caption, sample_rate). Optional kwargs are lists passed to each audio (ex: caption, sample_rate). z#Expected a list as "audios", found rrrcr)crr7r7rrr7rQrrz4WandbLogger.log_audio...r7rrrrQrrz)WandbLogger.log_audio..rNcrr7)Audio)raudiorrr7rQrrr) rNr|rrwr1rrrrrvr7rrQ log_audio  zWandbLogger.log_audiovideosc  r) aLog videos (numpy arrays, or file paths). Args: key: The key to be used for logging the video files videos: The list of video file paths, or numpy arrays to be logged step: The step number to be used for logging the video files **kwargs: Optional kwargs are lists passed to each Wandb.Video instance (ex: caption, fps, format). Optional kwargs are lists passed to each video (ex: caption, fps, format). z#Expected a list as "videos", found rrrcr)crr7r7rrr7rQrrz4WandbLogger.log_video...r7rrrrQrrz)WandbLogger.log_video..rNcrr7)Video)rvideorrr7rQr"rr) rNr|rrwr1rrrrrvr7rrQ log_video rzWandbLogger.log_videocC|jS)z`Gets the save directory. Returns: The path to the save directory. )rJrNr7r7rQr%%s zWandbLogger.save_dircCr)aThe project name of this experiment. Returns: The name of the project the current experiment belongs to. This name is not the same as `wandb.Run`'s name. To access wandb's internal experiment name, use ``logger.experiment.name`` instead. )rIrr7r7rQr$0s zWandbLogger.namecCs|jr|jjS|jS)zGets the id of the experiment. Returns: The id of the experiment if the experiment exists else the id given to the constructor. )r@r)rLrr7r7rQr&<s zWandbLogger.versioncheckpoint_callbackcCsD|jdks|jdur|jdkr||dS|jdur ||_dSdS)Nr-T)r> save_top_k_scan_and_log_checkpointsrB)rNrr7r7rQafter_save_checkpointHs   z!WandbLogger.after_save_checkpointartifact artifact_type use_artifactcCs\ddl}|jdur|r|j|}n |}|j||d}|dur#dnt|}|j|dS)aDownloads an artifact from the wandb server. Args: artifact: The path of the artifact to download. save_dir: The directory to save the artifact to. artifact_type: The type of artifact to download. use_artifact: Whether to add an edge between the artifact graph. Returns: The path to the downloaded artifact. rNr)root)rTrfrApirrCrDdownload)rr%rrrTapir7r7rQdownload_artifactPs zWandbLogger.download_artifactrcCs|jj||dS)aLogs to the wandb dashboard that the mentioned artifact is used by the run. Args: artifact: The path of the artifact. artifact_type: The type of artifact being used. Returns: wandb Artifact object for the artifact. r)r.r)rNrrr7r7rQros zWandbLogger.use_artifactstatuscCs4|dkrdS|jr|jdur||jdSdSdS)Nsuccess)rBr@r)rNrr7r7rQfinalize|s zWandbLogger.finalizec sddl}t|j}|D]W\}}}}dt|tr|n|dt|jjj fdddDi}|j s:d|j j |_ |j |j d|d } | j|d d |jkrSd d gnd g} |j j| | d||j|<q dS)Nrscoreoriginal_filenamecs"i|] }t|r|t|qSr7)rgrXrrr7rQrs   z9WandbLogger._scan_and_log_checkpoints..)monitormode save_lastrsave_weights_only_every_n_train_stepszmodel-rm)r$rmetadataz model.ckpt)r$latestbest)aliases)rTrrArir itemrr$rP__name__rMr.r)radd_filebest_model_path log_artifact) rNrrT checkpointstpstagrrrr7rrQrs"   z%WandbLogger._scan_and_log_checkpoints) Nr"NFNNNNFNr#N)rkrlT)N)NNNN)NNT).r __module__ __qualname____doc__ryr r:rboolr rrr<rr[propertyrr.nnModuleintrqrrrrur floatr{rrrrrrr%r$r&rr staticmethodrrrr __classcell__r7r7rOrQr 2sp     ;, &(     ***    r )2rrCargparserpathlibrtypingrrrrrr r r torch.nnr lightning_utilities.core.importsr torchr typing_extensionsr!lightning.fabric.utilities.loggerrrrr lightning.fabric.utilities.typesr,lightning.pytorch.callbacks.model_checkpointr lightning.pytorch.loggers.loggerrr#lightning.pytorch.loggers.utilitiesr&lightning.pytorch.utilities.exceptionsr%lightning.pytorch.utilities.rank_zerorrrTrrdrrerr8r r7r7r7rQs,   (