o ̳i.1@sddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZGd ddeZdS)N) HTTPStatus)Path)warn)snapshot_download)GatedRepoErrorRepositoryNotFoundError)model_download)set_kaggle_credentials)KaggleApiHTTPError)parse_model_handle) Subcommand) REPO_ID_FNAMEcseZdZdZdejffdd ZdddZd ejddfd d Z d ejddfd d Z d ejddfddZ de ddfddZ d ejfddZZS)Downloadz7Holds all the logic for the `tune download` subcommand. subparsersc sHt|jdddddtdtjd|_||jj |j ddS)Ndownloadz tune downloadz!tune download [OPTIONS]z?Download a model from the Hugging Face Hub or Kaggle Model Hub.a examples: # Download a model from the Hugging Face Hub with a Hugging Face API token $ tune download meta-llama/Llama-2-7b-hf --hf-token Successfully downloaded model repo and wrote to the following locations: /tmp/Llama-2-7b-hf/config.json /tmp/Llama-2-7b-hf/README.md /tmp/Llama-2-7b-hf/consolidated.00.pth ... # Download an ungated model from the Hugging Face Hub $ tune download mistralai/Mistral-7B-Instruct-v0.2 --output-dir /tmp/model Successfully downloaded model repo and wrote to the following locations: /tmp/model/config.json /tmp/model/README.md /tmp/model/model-00001-of-00002.bin ... # Download a model from the Kaggle Model Hub $ tune download metaresearch/llama-3.2/pytorch/1b --source kaggle Successfully downloaded model repo and wrote to the following locations: /tmp/llama-3.2/pytorch/1b/tokenizer.model /tmp/llama-3.2/pytorch/1b/params.json /tmp/llama-3.2/pytorch/1b/consolidated.00.pth ... For a list of all models, visit the Hugging Face Hub https://huggingface.co/models or Kaggle Model Hub https://kaggle.com/models. )progusagehelp descriptionepilogformatter_class)func) super__init__ add_parsertextwrapdedentargparseRawTextHelpFormatter_parser_add_arguments set_defaults _download_cmd)selfr __class__K/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/_cli/download.pyr s 'zDownload.__init__returnNcCs|jjdtdd|jjdtdddd|jjd tdtd dd d|jjd tdd d|jjdtddddgdd|jjdtddd|jjdtddddS)zAdd arguments to the parser.repo_idzOName of the repository on Hugging Face Hub or model handle on Kaggle Model Hub.)typerz --output-dirFNzFDirectory in which to save the model. Defaults to `/tmp/`.)r*requireddefaultrz --hf-tokenHF_TOKENz|j j tjtjhvrL|jdn|j j tjkr]|jd|dt}d |d |d |}|j|WYd }~d Sd }~wty}zt}d |d |d |}|j|WYd }~d Sd }~ww) z,Downloads a model from the Kaggle Model Hub.z]--output-dir flag is not supported for Kaggle model downloads. This argument will be ignored.zb--ignore-patterns flag is not supported for Kaggle model downloads. This argument will be ignored.rDrErFaSIt looks like you are trying to access a gated model. Please ensure you have access to the model and have provided the proper Kaggle credentials using the options `--kaggle-username` and `--kaggle-api-key`. You can also set these to environment variables as detailed in https://github.com/Kaggle/kagglehub/blob/main/README.md#authenticate.'z$' not found on the Kaggle Model Hub.r>r?r@N)r)_validate_kaggle_model_handle_set_kaggle_credentialsrGrr<rrIrTrrUr response status_coder UNAUTHORIZED FORBIDDENrrK NOT_FOUNDrMrNrL)r#r5 model_handlerGrXrYrZr&r&r'r8sT    zDownload._download_from_kagglehandlec Csz)t|}|jdkr|jdkrtd|d|jdvr'td|dWdSWdStyK}zd|d |d }|j|WYd}~dSd}~ww) Npytorch metaresearchzRequested PyTorch model zQ was not published from Meta, and therefore may not be compatible with torchtune.>rg transformerszRequested model zc is neither a PyTorch nor a Transformers model, and therefore may not be compatible with torchtune.zFailed to validate z with error .)r frameworkownerrrLrrK)r#rf parsed_handlerXrZr&r&r'r^s"      z&Download._validate_kaggle_model_handlec Csz-|js|jr+td|jr|jntjd}|jr|jntjd}t||WdSWdStyJ}zd|d}t|WYd}~dSd}~ww)NaTIP: you can avoid passing in the --kaggle-username and --kaggle-api-key arguments by storing them as the environment variables KAGGLE_USERNAME and KAGGLE_KEY, respectively. For more details, see https://github.com/Kaggle/kagglehub/blob/main/README.md#authenticateKAGGLE_USERNAME KAGGLE_KEYz.Failed to set Kaggle credentials with error: 'r]) kaggle_usernamekaggle_api_keyrIr3environgetr rLr)r#r5rprqrXrZr&r&r'r_s(     z Download._set_kaggle_credentials)r(N)__name__ __module__ __qualname____doc__r_SubParsersActionrr Namespacer"r7r8r2r^r_ __classcell__r&r&r$r'rs ,264r)rrRr3rrMhttprpathlibrwarningsrhuggingface_hubrhuggingface_hub.utilsrr kagglehubrkagglehub.authr kagglehub.exceptionsr kagglehub.handler torchtune._cli.subcommandr 'torchtune.training.checkpointing._utilsr rr&r&r&r's"