o ߥi0@sddlZddlmZddlmZmZmZddlmZm Z m Z ddl m Z ddl mZddlmZe ZGd d d ZGd d d ZdS) N)Optional)GitErrorInvalidParameterNotLoginException)DEFAULT_DATASET_REVISIONDEFAULT_REPOSITORY_REVISIONMASTER_MODEL_BRANCH) get_logger)GitCommandWrapper get_endpointc @seZdZdZeddfdededeedeedeef dd Zd d Zd d Z d%dedefddZ defddZ eedfdedeedeedee fddZ efdeded eefd!d"Zefdeded eefd#d$ZdS)& Repositoryz8A local representation of the model git repository. N model_dir clone_fromrevision auth_tokengit_pathc Cs ||_tj||_tj||_|sd}t|ddlm }|r%||_ n| |_ t }| s6tdt ||_tj|jdd||} t|jr`|} |j| } | r`| | kr`dS|j|j|j | |j|| rw||j|j|j|j|j r|j|j|j dSdS)a"Instantiate a Repository object by cloning the remote ModelScopeHub repo Args: model_dir (str): The model root directory. clone_from (str): model id in ModelScope-hub from which git clone revision (str, optional): revision of the model you want to clone from. Can be any of a branch, tag or commit hash auth_token (str, optional): token obtained when calling `HubApi.login()`. Usually you can safely ignore the parameter as the token is already saved when you login the first time, if None, we will use saved token. git_path (str, optional): The git command line path, if None, we use 'git' Raises: InvalidParameter: revision is None. 0a non-default value of revision cannot be empty.rModelScopeConfigz)git lfs is not installed, please install.Texist_okN)rospathdirnamemodel_base_dirbasenamemodel_repo_namermodelscope.hub.apirr get_tokenr is_lfs_installedloggererror git_wrappermakedirs_get_model_id_urllistdir_get_remote_urlremove_token_from_urlclonegit_lfs_install add_user_infoconfig_auth_token) selfrrrrrerr_msgrr$url remote_urlr2M/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/hub/repository.py__init__s@          zRepository.__init__cCstd|d}|S)N/.gitr )r.model_idr0r2r2r3r&PszRepository._get_model_id_urlcC.z |j|j}W|Styd}Y|SwN)r$get_repo_remote_urlrrr.remoter2r2r3r(T zRepository._get_remote_urloriginmasterr<branchcCs|jj|j||ddS)zPull remote branch Args: remote (str, optional): The remote name. Defaults to 'origin'. branch (str, optional): The remote branch. Defaults to 'master'. )r<r@N)r$pullr)r.r<r@r2r2r3rA[szRepository.pullfile_name_suffixcCs"td|tj|jdfdS)zAdd file suffix to lfs list. Args: file_name_suffix (str): The file name suffix. examples '*.safetensors' z5printf ' %s filter=lfs diff=lfs merge=lfs -text '>>%sz.gitattributesN)rsystemrjoinr)r.rBr2r2r3 add_lfs_typeds zRepository.add_lfs_typeFcommit_message local_branch remote_branchforcecCs|dus t|tsd}t|t|tstd|jstd|j|j|j|j |j |j |j |j}|jj |jdd|j|j||jj|j|j|||ddS)a Push local files to remote, this method will do. Execute git pull, git add, git commit, git push in order. Args: commit_message (str): commit message local_branch(str, optional): The local branch, default master. remote_branch (str, optional): The remote branch to push, default master. force (bool, optional): whether to use forced-push. Raises: InvalidParameter: no commit message. NotLoginException: no auth token. N commit_message must be provided!force must be bool'Must login to push, please login first.T all_filesrepo_dirtokenr0rGrH) isinstancestrrboolrrr$r-rr,rrr:addcommitpush)r.rFrGrHrImsgr0r2r2r3rWos*   zRepository.pushtag_namemessagerefcCsR|dus|dkrd}t||dus|dkrd}t||jj|j|||ddS)aCreate a new tag. Args: tag_name (str): The name of the tag message (str): The tag message. ref (str, optional): The tag reference, can be commit id or branch. Raises: InvalidParameter: no commit message. NzFWe use tag-based revision, therefore tag_name cannot be None or empty.z=We use annotated tag, therefore message cannot None or empty.)rPrYrZr[)rr$tagr)r.rYrZr[rXr2r2r3r]s zRepository.tagcCs$|||||jj|j|ddS)zCreate tag and push to remote Args: tag_name (str): The name of the tag message (str): The tag message. ref (str, optional): The tag ref, can be commit id or branch. Defaults to MASTER_MODEL_BRANCH. )rPrYN)r]r$push_tagr)r.rYrZr[r2r2r3 tag_and_pushs zRepository.tag_and_push)r>r?)__name__ __module__ __qualname____doc__rrSrr4r&r(rArErTrWrr]r_r2r2r2r3rs` ;   - rc @seZdZdZeddfdededeedeedeef dd Zd efd d Zed fdedeedee fddZ ddZ ddZ dS)DatasetRepositoryzEA local representation of the dataset (metadata) git repository. N repo_work_dir dataset_idrrrcCs||_|r t|tsd}t||d|_|jsd}t|tj|j|_ tj |j|_ |s7d}t|||_ ddl m}|rF||_n||_t||_tj|jdd|j|d |_d S) au Instantiate a Dataset Repository object by cloning the remote ModelScope dataset repo Args: repo_work_dir (str): The dataset repo root directory. dataset_id (str): dataset id in ModelScope from which git clone revision (str, optional): revision of the dataset you want to clone from. Can be any of a branch, tag or commit hash auth_token (str, optional): token obtained when calling `HubApi.login()`. Usually you can safely ignore the parameter as the token is already saved when you login the first time, if None, we will use saved token. git_path (str, optional): The git command line path, if None, we use 'git' Raises: InvalidParameter: parameter invalid. z"dataset_work_dir must be provided!r5z%dataset_work_dir can not be root dir!rrrTr)rfN)rfrRrSrrstriprerrr repo_base_dirr repo_namerrrrr r r$r% _get_repo_urlrepo_url)r.rerfrrrr/rr2r2r3r4s*    zDatasetRepository.__init__returncCsht|jr|}|j|}|r||jkrdStd |j|j |j |j |j|j |j|jS)Nr\zCloning repo from {} )rr'rer(r$r)rkr"infoformatr*rhrrir)r.r1r2r2r3r*s   zDatasetRepository.cloneFrFr@rIcCs|dus t|tsd}t|t|tstd|jstd|j|j|j|j |j |j | }|j |}|j|j|jj|jdd|j|j||jj|j|j|||ddS)aPush local files to remote, this method will do. git pull git add git commit git push Args: commit_message (str): commit message branch (str, optional): which branch to push. force (bool, optional): whether to use forced-push. Raises: InvalidParameter: no commit message. NotLoginException: no access token. NrJrKrLTrMrO)rRrSrrTrrr$r-rer,rhrir(r)rArUrVrW)r.rFr@rIrXr1r2r2r3rWs*   zDatasetRepository.pushcCstd|dS)Nz /datasets/r6r )r.rfr2r2r3rj1szDatasetRepository._get_repo_urlcCr8r9)r$r:rerr;r2r2r3r(4r=z!DatasetRepository._get_remote_url) r`rarbrcrrSrr4r*rTrWrjr(r2r2r2r3rds8 / - rd)rtypingrmodelscope.hub.errorsrrrmodelscope.utils.constantrrrmodelscope.utils.loggerr gitr utils.utilsr r"rrdr2r2r2r3s    3