o
si2 @ s d dl Z d dlZd dlZd dlmZ dZdZdZdZeddZ d d d
Z
d d d
Z d ddZ dd Z
d ddZd ddZdd Zdd Zdd ZdS ) N)pprintz
Don't forget to share your pretrained models at https://zenodo.org/communities/asteroid-models/! =)
You can directly use our CLI for that, run this:
`asteroid-upload {} --uploader "Your name here"`
z{}z#Attribution-ShareAlike 3.0 Unportedz/https://creativecommons.org/licenses/by-sa/3.0/z)https://github.com/asteroid-team/asteroidAsteroidc C s d| v s
J dd| v sJ dd| v s J dd| v s(J dt|ts1J d |d
urAt|ts>J d |}n tjtj| d r^tt tj| d }|
d
d nd}||d d< ||d d< ||d d< tj | dd t
|tj| d tt| |S )a Save models to prepare for publication / model sharing.
Args:
publish_dir (str): Path to the publishing directory.
Usually under exp/exp_name/publish_dir
model_dict (dict): dict at least with keys `model_args`,
`state_dict`,`dataset` or `licenses`
metrics (dict): dict with evaluation metrics.
train_conf (dict): Training configuration dict (from conf.yml).
recipe (str): Name of the recipe.
Returns:
dict, same as `model_dict` with added fields.
Raises:
AssertionError when either `model_args`, `state_dict`,`dataset` or
`licenses` are not present is `model_dict.keys()`
model_argsz%`model_args` not found in model dict.
state_dictz%`state_dict` not found in model dict.datasetz"`dataset` not found in model dict.licensesz#`licenses` not found in model dict.z&Cannot upload a model without metrics.Nz `recipe` should be a string.zrecipe_name.txt
Unknowninfos recipe_nametraining_config
final_metricsT)exist_ok model.pth)keys
isinstancedictstrospathexistsjoinnextopenreplacemakedirstorchsaveprintPLEASE_PUBLISHformat) publish_dir
model_dictmetrics
train_confreciper r' M/home/ubuntu/.local/lib/python3.10/site-packages/asteroid/models/publisher.pysave_publishable s& r) Fc sF fdd |du rt dtj| d}tj| d} t|}
t|
|||d}
t|
| |du r@td}|du r@t d t |
|| |d
\} } tj| j
d t | }
|rk| | }t
| td |
|S | | }td
|
d t
| |r| |fS }|dkr| | }td|
dS td|
d dS )a Entry point to upload publishable model.
Args:
publish_dir (str): Path to the publishing directory.
Usually under exp/exp_name/publish_dir
uploader (str): Full name of the uploader (Ex: Manuel Pariente)
affiliation (str, optional): Affiliation (no accent).
git_username (str, optional): GitHub username.
token (str): Access token generated to upload depositions.
force_publish (bool): Whether to directly publish without
asking confirmation before. Defaults to False.
use_sandbox (bool): Whether to use Zenodo's sandbox instead of
the official Zenodo.
unit_test (bool): If True, we do not ask user input and do not publish.
c s* t d} | dvrtd| d S | S )NzE
Do you want to publish it now (irreversible)? y/n(Recommended: n).
)ynz&
Expected one of [`y`, `n`], received z, please retry.)inputr )out
get_answerr' r( r/ Z s
z&upload_publishable..get_answerNzNeed uploader namer zpublished_model.pth)uploader affiliation git_username ACCESS_TOKENaa Need an access token to Zenodo to upload the model. Either set ACCESS_TOKEN environment variable or pass it directly (`asteroid-upload --token ...`).If you do not have a access token, first create a Zenodo account (https://zenodo.org/signup/), create a token https://zenodo.org/account/settings/applications/tokens/new/and you are all set to help us! =))
model_path use_sandboxdepositzYou can also visit it at {}z9
This is the current state of the deposition (see here z): r* zVisit it at {}z*Did not finalize the upload, please visit z to finalize it.)
ValueErrorr r r r load_populate_publishabler getenv
zenodo_uploadzenodo_addressr publish_depositionr jsonr r! get_deposition)r" r0 r1 r2 token
force_publishr5 unit_testr4 publish_model_pathmodelzendep_idaddress r_publishcurrentinp_r' r. r( upload_publishable? sL
rL c C s |du rt }d| d | d | d ddg}|d | }t|| d |d }|| d
d < || d
d < |r8|nd
| d
d< || d
d< || d
d< | S )a Populate infos in publishable model.
Args:
model (dict): Model to publish, with `infos` key, at least.
uploader (str): Full name of the uploader (Ex: Manuel Pariente)
affiliation (str, optional): Affiliation (no accent).
git_username (str, optional): GitHub username.
Returns:
dict (model), same as input `model`
.. note:: If a `git_username` is not specified, we look for it somehow, or take
the laptop username.
NrK
model_namer taskr /r )r0 r r0 r2 r
r1 upload_name license_note) get_usernamer r make_license_notice)rD r0 r1 r2 rM rP rQ r' r' r( r9 s $ r9 c C s8 t g d} | ddd } | sddl}| } | S )z"Get git of FS username for upload.)gitconfigz user.namezutf-8Nr )
subprocess check_outputdecodegetpassgetuser)usernamerZ r' r' r( rR s rR c
C s |du rt dd| }|D ];}t|d |d }t|d |d }t|d |d }|d
r5d nd }|d
| d| d| | 7 }|d7 }q|dd d }ttt} |d| d| d| d7 }|S )a Make license notice based on license dicts.
Args:
model_name (str): Name of the model.
licenses (List[dict]): List of dict with
keys (`title`, `title_link`, `author`, `author_link`,
`licence`, `licence_link`).
uploader (str): Name of the uploader such as "Manuel Pariente".
Returns:
str, the license note describing the model, it's attribution,
the original licenses, what we license it under and the licensor.
Nz$Cannot share model without uploader.z This work "{}" is a derivative
title_linktitle author_linkauthor license_linklicensenon_commercialz (Research only)r zof z by z
, used under z; z. "z" is licensed under .)r7 r! HREF
CC_SA_LINKCC_SA)
rM r r0 notel_dictr^ r` license_hcommcc_sar' r' r( rS s
rS c C s d}|du rd}d}t | | ddlm} |||d}t| }|j|d}|jd kr5t| t d
| d } |j
| |d d
}
|rJt | || fS )a Create deposit and upload metadata + model
Args:
model (dict):
token (str): Access token.
model_path (str): Saved model path.
use_sandbox (bool): Whether to use Zenodo's sandbox instead of
the official Zenodo.
Returns:
Zenodo (Zenodo instance with access token)
int (deposit ID)
.. note::If `model_path` is not specified, save the model in tmp.pth and
remove it after upload.
FNTztmp.pth )Zenodo)r5 )metadata z:Could not create the deposition, check the provided token.idr )name)
r r zenodorp make_metadata_from_modelcreate_new_deposition status_coder r> RuntimeError upload_new_file_to_depositionr remove) rD r@ r4 r5 model_path_was_nonerp rE rq rrF rK r' r' r( r; s"
r; c C s$ | d }d}d}|| |d |d t7 }d}|| | d | d 7 }|d 7 }|d
7 }|t|d 7 }|d 7 }|d 7 }d
d |d D }|t|7 }|d 7 }|d7 }|t|d 7 }|d 7 }|d7 }||d 7 }|d d||d |d dgddiddigdd | d | d | d d gd d }|S )!a^ Create Zenodo deposit metadata for a given publishable model.
Args:
model (dict): Dictionary with all infos needed to publish.
More info to come.
Returns:
dict, the metadata to create the Zenodo deposit with.
.. note:: We remove the PESQ from the final results as a license is needed to
use it.
r z%Description:
z8This model was trained by {} using the {} recipe in {}. r0 r zEIt was trained on the {} task of the {} dataset.rN r z
z(Training config:
r
z Results:
c S s" i | ]
\}}d | vr||qS )pesq)lower).0kvr' r' r(
' s " z,make_metadata_from_model..r z!Versions:
software_versionsz'License notice:
rQ rP softwarer1 )rt r1
identifierru zasteroid-modelsr zaudio source separationrM zpretrained modelz CC-BY-SA-3.0)r^ upload_type descriptioncreators communitieskeywordsrb )r! ASTEROID_REFtwo_level_dict_htmlitemsdisplay_one_level_dict)rD r r tmpdisplay_resultrq r' r' r( rv
sD
rv c C sn d}| D ]*}|d| d7 }| | D ]}t| | | }|d| d| d7 }q|d7 }q|d7 }|S )zTwo-level dict to HTML.
Args:
dic (dict): two-level dict
Returns:
str for HTML-encoded two level dic
r r )dichtmlr k2valr' r' r( r H s
r c C s@ d}| D ]}t| | }|d| d| d7 }q|d7 }|S )zySingle level dict to HTML
Args:
dic (dict):
Returns:
str for HTML-encoded single level dic
r r r z r r )r r r r r' r' r( r ^ s r )NNN)NNNNFFF)N)NF)r r rW r r rg ri rh r! r r) rL r9 rR rS r; rv r r r' r' r' r( s4
-
Z!
&>