o piKi@sddlZddlmZddlmZmZmZddlZddlm Z ddl Z ddl mZmZddlmZmZddlmZmZddlmZmZddlmZmZmZmZdd lmZd d l m!Z!m"Z"d d l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)d dl*m+Z+e),e-Z.Gddde(Z/dS)N)UnpicklingError)AnyDictUnion) FrozenDictunfreeze) from_bytesto_bytes) flatten_dictunflatten_dict) create_repohf_hub_download)EntryNotFoundErrorRepositoryNotFoundErrorRevisionNotFoundErrorvalidate_hf_hub_args) HTTPError) __version__is_torch_available) CONFIG_NAMEFLAX_WEIGHTS_NAMEHUGGINGFACE_CO_RESOLVE_ENDPOINT WEIGHTS_NAMEPushToHubMixinlogging)"convert_pytorch_state_dict_to_flaxc @seZdZdZeZgdZgdZeddZ d de e e fde jd ed efd d Zd de e e fd efd dZd de e e fd efddZd de e e fd efddZdejd e fddZeee jfde eejfde jfddZ  d!de eejfde e e fdedefddZdS)"FlaxModelMixina* Base class for all Flax models. [`FlaxModelMixin`] takes care of storing the model configuration and provides methods for loading, downloading and saving models. - **config_name** ([`str`]) -- Filename to save a model to when calling [`~FlaxModelMixin.save_pretrained`]. )_diffusers_version _class_name _name_or_path)nameparentdtypecKs||fi|S)zZ All context managers that the model should be initialized under go here. )clsconfigkwargsr%r%b/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/models/modeling_flax_utils.py _from_config@szFlaxModelMixin._from_configNparamsr$maskreturnc snfdd}|durt||St|}t|\}}t||D]\}} |r2|| } || || <q"t|S)zk Helper method to cast floating-point values of given parameter `PyTree` to given `dtype`. cs*t|tjrt|jtjr|}|SN) isinstancejnpndarray issubdtyper$floatingastype)paramr$r%r)conditional_castMs z:FlaxModelMixin._cast_floating_to..conditional_castN)jaxtree_mapr tree_flattenzipkeysr ) selfr+r$r,r7 flat_params flat_mask_maskedkeyr5r%r6r)_cast_floating_toGs   z FlaxModelMixin._cast_floating_tocC||tj|S)a Cast the floating-point `params` to `jax.numpy.bfloat16`. This returns a new `params` tree and does not cast the `params` in place. This method can be used on a TPU to explicitly convert the model parameters to bfloat16 precision to do full half-precision training or to save weights in bfloat16 for inference in order to save memory and improve speed. Arguments: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. mask (`Union[Dict, FrozenDict]`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans. It should be `True` for params you want to cast, and `False` for those you want to skip. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # load model >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> # By default, the model parameters will be in fp32 precision, to cast these to bfloat16 precision >>> params = model.to_bf16(params) >>> # If you don't want to cast certain parameters (for example layer norm bias and scale) >>> # then pass the mask as follows >>> from flax import traverse_util >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> flat_params = traverse_util.flatten_dict(params) >>> mask = { ... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) ... for path in flat_params ... } >>> mask = traverse_util.unflatten_dict(mask) >>> params = model.to_bf16(params, mask) ```)rCr0bfloat16r=r+r,r%r%r)to_bf16_%zFlaxModelMixin.to_bf16cCrD)a Cast the floating-point `params` to `jax.numpy.float32`. This method can be used to explicitly convert the model parameters to fp32 precision. This returns a new `params` tree and does not cast the `params` in place. Arguments: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. mask (`Union[Dict, FrozenDict]`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans. It should be `True` for params you want to cast, and `False` for those you want to skip. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # Download model and configuration from huggingface.co >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> # By default, the model params will be in fp32, to illustrate the use of this method, >>> # we'll first cast to fp16 and back to fp32 >>> params = model.to_f16(params) >>> # now cast back to fp32 >>> params = model.to_fp32(params) ```)rCr0float32rFr%r%r)to_fp32szFlaxModelMixin.to_fp32cCrD)a Cast the floating-point `params` to `jax.numpy.float16`. This returns a new `params` tree and does not cast the `params` in place. This method can be used on a GPU to explicitly convert the model parameters to float16 precision to do full half-precision training or to save weights in float16 for inference in order to save memory and improve speed. Arguments: params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. mask (`Union[Dict, FrozenDict]`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans. It should be `True` for params you want to cast, and `False` for those you want to skip. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # load model >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> # By default, the model params will be in fp32, to cast these to float16 >>> params = model.to_fp16(params) >>> # If you want don't want to cast certain parameters (for example layer norm bias and scale) >>> # then pass the mask as follows >>> from flax import traverse_util >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> flat_params = traverse_util.flatten_dict(params) >>> mask = { ... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) ... for path in flat_params ... } >>> mask = traverse_util.unflatten_dict(mask) >>> params = model.to_fp16(params, mask) ```)rCr0float16rFr%r%r)to_fp16rHzFlaxModelMixin.to_fp16rngcCstd|)Nz.init_weights method has to be implemented for )NotImplementedError)r=rMr%r%r) init_weightsszFlaxModelMixin.init_weightspretrained_model_name_or_pathc" Osj|dd}|dd}|dd}|dd}|dd} |dd} |d d} |d d} |d d} td d d}|durT|j|f|d|| | | | | d|\}}|j|f|dd|\}}| durh|ntj|| }tj|r|rtjtj|t st dt d|dtj|t }ntjtj|t rtj|t }ntjtj|t rt t d|dt dt dt d|dzt ||st nt ||| | | || | d }Wnut yt |dtyt | d|dtyt |dt dty$}z t d|d|d}~wty<t d td!|d"t dt d# t yTt d$|d%|d&t dt d w|rqtrcd'd(lm}nt d)||}t||}nizt|d*}t||}Wdn 1swYWnIttjjfy}z8zt|}|d+rtd,t|1swYWnt tfyt d-|d.wWYd}~nd}~wwt!j"#d/d0|}t$|}t!j%|j&t!j'(d1d2}t)t$t*|+}t$t*|}|t)|+}t)|+|}|r%t,-d3|d4|d5||_.|+D]'} | |vrO|| j/|| j/krOtd6| d7|| j/d8|| j/d9q)|D]}!||!=qSt0|d1krxt,-d:|d;|j1j2d<|d=|j1j2d> n t,3d?|j1j2d@t0|d1krt,-dA|j1j2dB|dC|dDnt,3dE|j1j2dF|dG|j1j2dH|t4|fS)Ia Instantiate a pretrained Flax model from a pretrained model configuration. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* (for example `runwayml/stable-diffusion-v1-5`) of a pretrained model hosted on the Hub. - A path to a *directory* (for example `./my_model_directory`) containing the model weights saved using [`~FlaxModelMixin.save_pretrained`]. dtype (`jax.numpy.dtype`, *optional*, defaults to `jax.numpy.float32`): The data type of the computation. Can be one of `jax.numpy.float32`, `jax.numpy.float16` (on GPUs) and `jax.numpy.bfloat16` (on TPUs). This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If specified, all the computation will be performed with the given `dtype`. This only specifies the dtype of the *computation* and does not influence the dtype of model parameters. If you wish to change the dtype of the model parameters, see [`~FlaxModelMixin.to_fp16`] and [`~FlaxModelMixin.to_bf16`]. model_args (sequence of positional arguments, *optional*): All remaining positional arguments are passed to the underlying model's `__init__` method. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory where a downloaded pretrained model configuration is cached if the standard cache is not used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, for example, `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether to only load local model weights and configuration files or not. If set to `True`, the model won't be downloaded from the Hub. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, a commit id, or any identifier allowed by Git. from_pt (`bool`, *optional*, defaults to `False`): Load the model weights from a PyTorch checkpoint save file. kwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it is loaded) and initiate the model (for example, `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `kwargs` are directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done). - If a configuration is not provided, `kwargs` are first passed to the configuration class initialization function [`~ConfigMixin.from_config`]. Each key of the `kwargs` that corresponds to a configuration attribute is used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute are passed to the underlying model's `__init__` function. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # Download model and configuration from huggingface.co and cache. >>> model, params = FlaxUNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5") >>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable). >>> model, params = FlaxUNet2DConditionModel.from_pretrained("./test/saved_model/") ``` If you get the error message below, you need to finetune the weights for your downstream task: ```bash Some weights of UNet2DConditionModel were not initialized from the model checkpoint at runwayml/stable-diffusion-v1-5 and are newly initialized because the shapes did not match: - conv_in.weight: found shape torch.Size([320, 4, 3, 3]) in the checkpoint and torch.Size([320, 9, 3, 3]) in the model instantiated You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. ``` r'N cache_dirforce_downloadFfrom_ptproxieslocal_files_onlytokenrevision subfoldermodelflax) diffusers file_type frameworkT)rQreturn_unused_kwargsrRrTrUrVrWrX)r$r^zError no file named z found in directory  z file found in directory z-. Please load the model using `from_pt=True`.z or .) filenamerQrRrTrUrV user_agentrXrWz is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo with `token` or log in with `huggingface-cli login`.z is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at 'https://huggingface.co/z' for available revisions.z& does not appear to have a file named z:There was a specific connection error when trying to load z: zWe couldn't connect to 'zM' to load this model, couldn't find it in the cached files and it looks like z8 is not the path to a directory containing a file named z. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named r)load_state_dictz|Can't load the model in PyTorch format because PyTorch is not installed. Please, install PyTorch or use native Flax weights.rbversionzYou seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.zUnable to convert z to Flax deserializable object. cSst|tjdddS)Ncpu)backendr)r8 device_put local_devices)xr%r%r)sz0FlaxModelMixin.from_pretrained..r)rMzThe checkpoint z is missing required keys: zI. Make sure to call model.init_weights to initialize the missing weights.z)Trying to load the pretrained weight for z failed: checkpoint has shape z, which is incompatible with the model shape z. z(Some weights of the model checkpoint at z! were not used when initializing z: z, - This IS expected if you are initializing zU from the checkpoint of a model trained on another task or with another architecture.z9All model checkpoint weights were used when initializing z. zSome weights of z3 were not initialized from the model checkpoint at z and are newly initialized: zo You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.zAll the weights of z/ were initialized from the model checkpoint at zf. If your task is similar to the task the model of the checkpoint was trained on, you can already use z* for predictions without further training.)5popr load_config from_configospathjoinisdirisfilerEnvironmentErrorrr rrrr ValueErrorrrmodeling_utilsrcropenrreadrmsgpack exceptions ExtraData startswithOSErrorUnicodeDecodeErrorr8 tree_utilr9r eval_shaperOrandomPRNGKeysetrr<loggerwarning _missing_keysshapelen __class____name__infor )"r&rPr$ model_argsr(r'rQrRrSrTrUrVrWrXrb unused_kwargsrY model_kwargspretrained_path_with_subfolder model_fileerrrcpytorch_model_filestatestate_fefparams_shape_treerequired_params shape_state missing_keysunexpected_keysrBunexpected_keyr%r%r)from_pretrainedsr Z                           zFlaxModelMixin.from_pretrainedTFsave_directoryis_main_process push_to_hubcKstj|rtd|ddStj|dd|rH|dd}|dd}|d d}|d d} |d |tjjd } t | d|| d j } |} |rQ| |tj |t } t| d} t|}| |Wdn1sqwYtd| |r|j|| | ||ddSdS)a Save a model and its configuration file to a directory so that it can be reloaded using the [`~FlaxModelMixin.from_pretrained`] class method. Arguments: save_directory (`str` or `os.PathLike`): Directory to save a model and its configuration file to. Will be created if it doesn't exist. params (`Union[Dict, FrozenDict]`): A `PyTree` of model parameters. is_main_process (`bool`, *optional*, defaults to `True`): Whether the process calling this is the main process or not. Useful during distributed training and you need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions. push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. zProvided path (z#) should be a directory, not a fileNT)exist_okcommit_messageprivateF create_prrVrepo_id)rrrVwbzModel weights saved in )rVrr)rorprsrerrormakedirsrlsplitsepr r save_configrqrrwr writer_upload_folder)r=rr+rrr(rrrrVr model_to_saveoutput_model_filer model_bytesr%r%r)save_pretraineds:         zFlaxModelMixin.save_pretrainedr.)TF)r __module__ __qualname____doc__r config_name_automatically_saved_args_flax_internal_args classmethodr*rrrr0r$rrCrGrJrLr8ArrayrOrrIstrroPathLikerboolrr%r%r%r)r2sB  &'' *  r)0ropicklertypingrrrr8 jax.numpynumpyr0msgpack.exceptionsryflax.core.frozen_dictrrflax.serializationrr flax.traverse_utilr r huggingface_hubr r huggingface_hub.utilsrrrrrequestsrrrutilsrrrrrrmodeling_flax_pytorch_utilsr get_loggerrrrr%r%r%r)s"