o ۷ii@sddlZddlmZddlmZddlZddlmZddl Z ddl m Z m Z ddlmZmZddlmZmZddlmZmZddlmZmZmZmZmZd d lmZmZd d l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&d d l'm(Z(e&)e*Z+Gddde%Z,dS)N)UnpicklingError)Any) FrozenDictunfreeze) from_bytesto_bytes) flatten_dictunflatten_dict) create_repohf_hub_download)EntryNotFoundErrorHfHubHTTPErrorRepositoryNotFoundErrorRevisionNotFoundErrorvalidate_hf_hub_args) __version__is_torch_available) CONFIG_NAMEFLAX_WEIGHTS_NAMEHUGGINGFACE_CO_RESOLVE_ENDPOINT WEIGHTS_NAMEPushToHubMixinlogging)"convert_pytorch_state_dict_to_flaxc @seZdZdZeZgdZgdZeddZ d de e Bde j d ed efd d Zd de e Bd efd dZd de e Bd efddZd de e Bd efddZdejd e fddZeee jfdeejBde j fddZ  d!deejBde e BdedefddZdS)"FlaxModelMixina* Base class for all Flax models. [`FlaxModelMixin`] takes care of storing the model configuration and provides methods for loading, downloading and saving models. - **config_name** ([`str`]) -- Filename to save a model to when calling [`~FlaxModelMixin.save_pretrained`]. )_diffusers_version _class_name _name_or_path)nameparentdtypecKs||fi|S)zZ All context managers that the model should be initialized under go here. )clsconfigkwargsr#r#Z/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/models/modeling_flax_utils.py _from_config@szFlaxModelMixin._from_configNparamsr"maskreturnc snfdd}|durt||St|}t|\}}t||D]\}} |r2|| } || || <q"t|S)zk Helper method to cast floating-point values of given parameter `PyTree` to given `dtype`. cs*t|tjrt|jtjr|}|SN) isinstancejnpndarray issubdtyper"floatingastype)paramr"r#r'conditional_castMs z:FlaxModelMixin._cast_floating_to..conditional_castN)jaxtree_mapr tree_flattenzipkeysr ) selfr)r"r*r5 flat_params flat_mask_maskedkeyr3r#r4r'_cast_floating_toGs   z FlaxModelMixin._cast_floating_tocC||tj|S)a Cast the floating-point `params` to `jax.numpy.bfloat16`. This returns a new `params` tree and does not cast the `params` in place. This method can be used on a TPU to explicitly convert the model parameters to bfloat16 precision to do full half-precision training or to save weights in bfloat16 for inference in order to save memory and improve speed. Arguments: params (`dict | FrozenDict`): A `PyTree` of model parameters. mask (`dict | FrozenDict`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans. It should be `True` for params you want to cast, and `False` for those you want to skip. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # load model >>> model, params = FlaxUNet2DConditionModel.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5") >>> # By default, the model parameters will be in fp32 precision, to cast these to bfloat16 precision >>> params = model.to_bf16(params) >>> # If you don't want to cast certain parameters (for example layer norm bias and scale) >>> # then pass the mask as follows >>> from flax import traverse_util >>> model, params = FlaxUNet2DConditionModel.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5") >>> flat_params = traverse_util.flatten_dict(params) >>> mask = { ... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) ... for path in flat_params ... } >>> mask = traverse_util.unflatten_dict(mask) >>> params = model.to_bf16(params, mask) ```)rAr.bfloat16r;r)r*r#r#r'to_bf16_%zFlaxModelMixin.to_bf16cCrB)a Cast the floating-point `params` to `jax.numpy.float32`. This method can be used to explicitly convert the model parameters to fp32 precision. This returns a new `params` tree and does not cast the `params` in place. Arguments: params (`dict | FrozenDict`): A `PyTree` of model parameters. mask (`dict | FrozenDict`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans. It should be `True` for params you want to cast, and `False` for those you want to skip. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # Download model and configuration from huggingface.co >>> model, params = FlaxUNet2DConditionModel.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5") >>> # By default, the model params will be in fp32, to illustrate the use of this method, >>> # we'll first cast to fp16 and back to fp32 >>> params = model.to_f16(params) >>> # now cast back to fp32 >>> params = model.to_fp32(params) ```)rAr.float32rDr#r#r'to_fp32szFlaxModelMixin.to_fp32cCrB)a Cast the floating-point `params` to `jax.numpy.float16`. This returns a new `params` tree and does not cast the `params` in place. This method can be used on a GPU to explicitly convert the model parameters to float16 precision to do full half-precision training or to save weights in float16 for inference in order to save memory and improve speed. Arguments: params (`dict | FrozenDict`): A `PyTree` of model parameters. mask (`dict | FrozenDict`): A `PyTree` with same structure as the `params` tree. The leaves should be booleans. It should be `True` for params you want to cast, and `False` for those you want to skip. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # load model >>> model, params = FlaxUNet2DConditionModel.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5") >>> # By default, the model params will be in fp32, to cast these to float16 >>> params = model.to_fp16(params) >>> # If you want don't want to cast certain parameters (for example layer norm bias and scale) >>> # then pass the mask as follows >>> from flax import traverse_util >>> model, params = FlaxUNet2DConditionModel.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5") >>> flat_params = traverse_util.flatten_dict(params) >>> mask = { ... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) ... for path in flat_params ... } >>> mask = traverse_util.unflatten_dict(mask) >>> params = model.to_fp16(params, mask) ```)rAr.float16rDr#r#r'to_fp16rFzFlaxModelMixin.to_fp16rngcCstd|)Nz.init_weights method has to be implemented for )NotImplementedError)r;rKr#r#r' init_weightsszFlaxModelMixin.init_weightspretrained_model_name_or_pathc" Osvtd|dd}|dd}|dd}|dd}|dd} |d d} |d d} |d d} |d d} td dd}|durY|j|f|d|| | | | | d|\}}|j|f|dd|\}}| durm|ntj|| }tj |r|rtj tj|t st dt d|dtj|t }ntj tj|t rtj|t }ntj tj|t rt t d|dt dt dt d|dzt||st nt ||| | | || | d }Wnvtyt |dtyt | d|dtyt |dt dty*}z t d|d |d}~wtyBt d!td"|d#t dt d$ t yZt d%|d&|d't dt d w|rwtrid(d)lm}nt d*||}t||}nizt|d+}t||}Wdn 1swYWnIttjjfy}z8zt|}| d,rt!d-t|1swYWnt"tfyt d.|d/wWYd}~nd}~wwt#j$%d0d1|}t&|}t#j'|j(t#j)*d2d3}t+t&t,|-}t&t,|}|t+|-}t+|-|}|r+td4|d5|d6||_.|-D]'} | |vrU|| j/|| j/krUtd7| d8|| j/d9|| j/d:q/|D]}!||!=qYt0|d2kr~td;|d<|j1j2d=|d>|j1j2d? n t3d@|j1j2dAt0|d2krtdB|j1j2dC|dD|dEnt3dF|j1j2dG|dH|j1j2dI|t4|fS)Ja Instantiate a pretrained Flax model from a pretrained model configuration. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* (for example `stable-diffusion-v1-5/stable-diffusion-v1-5`) of a pretrained model hosted on the Hub. - A path to a *directory* (for example `./my_model_directory`) containing the model weights saved using [`~FlaxModelMixin.save_pretrained`]. dtype (`jax.numpy.dtype`, *optional*, defaults to `jax.numpy.float32`): The data type of the computation. Can be one of `jax.numpy.float32`, `jax.numpy.float16` (on GPUs) and `jax.numpy.bfloat16` (on TPUs). This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If specified, all the computation will be performed with the given `dtype`. > [!TIP] > This only specifies the dtype of the *computation* and does not influence the dtype of model > parameters. > > If you wish to change the dtype of the model parameters, see [`~FlaxModelMixin.to_fp16`] and > [`~FlaxModelMixin.to_bf16`]. model_args (sequence of positional arguments, *optional*): All remaining positional arguments are passed to the underlying model's `__init__` method. cache_dir (`str | os.PathLike`, *optional*): Path to a directory where a downloaded pretrained model configuration is cached if the standard cache is not used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, for example, `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether to only load local model weights and configuration files or not. If set to `True`, the model won't be downloaded from the Hub. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, a commit id, or any identifier allowed by Git. from_pt (`bool`, *optional*, defaults to `False`): Load the model weights from a PyTorch checkpoint save file. kwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it is loaded) and initiate the model (for example, `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `kwargs` are directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done). - If a configuration is not provided, `kwargs` are first passed to the configuration class initialization function [`~ConfigMixin.from_config`]. Each key of the `kwargs` that corresponds to a configuration attribute is used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute are passed to the underlying model's `__init__` function. Examples: ```python >>> from diffusers import FlaxUNet2DConditionModel >>> # Download model and configuration from huggingface.co and cache. >>> model, params = FlaxUNet2DConditionModel.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5") >>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable). >>> model, params = FlaxUNet2DConditionModel.from_pretrained("./test/saved_model/") ``` If you get the error message below, you need to finetune the weights for your downstream task: ```bash Some weights of UNet2DConditionModel were not initialized from the model checkpoint at stable-diffusion-v1-5/stable-diffusion-v1-5 and are newly initialized because the shapes did not match: - conv_in.weight: found shape torch.Size([320, 4, 3, 3]) in the checkpoint and torch.Size([320, 9, 3, 3]) in the model instantiated You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. ``` zFlax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.r%N cache_dirforce_downloadFfrom_ptproxieslocal_files_onlytokenrevision subfoldermodelflax) diffusers file_type frameworkT)rOreturn_unused_kwargsrPrRrSrTrUrV)r"r\zError no file named z found in directory  z file found in directory z-. Please load the model using `from_pt=True`.z or .) filenamerOrPrRrSrT user_agentrVrUz is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo with `token` or log in with `hf auth login`.z is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at 'https://huggingface.co/z' for available revisions.z& does not appear to have a file named z:There was a specific connection error when trying to load z: zWe couldn't connect to 'zM' to load this model, couldn't find it in the cached files and it looks like z8 is not the path to a directory containing a file named z. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named r)load_state_dictz|Can't load the model in PyTorch format because PyTorch is not installed. Please, install PyTorch or use native Flax weights.rbversionzYou seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run `git lfs install` followed by `git lfs pull` in the folder you cloned.zUnable to convert z to Flax deserializable object. cSst|tjdddS)Ncpu)backendr)r6 device_put local_devices)xr#r#r'sz0FlaxModelMixin.from_pretrained..r)rKzThe checkpoint z is missing required keys: zI. Make sure to call model.init_weights to initialize the missing weights.z)Trying to load the pretrained weight for z failed: checkpoint has shape z, which is incompatible with the model shape z. z(Some weights of the model checkpoint at z! were not used when initializing z: z, - This IS expected if you are initializing zU from the checkpoint of a model trained on another task or with another architecture.z9All model checkpoint weights were used when initializing z. zSome weights of z3 were not initialized from the model checkpoint at z and are newly initialized: zo You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.zAll the weights of z/ were initialized from the model checkpoint at zf. If your task is similar to the task the model of the checkpoint was trained on, you can already use z* for predictions without further training.)5loggerwarningpopr load_config from_configospathjoinisdirisfilerEnvironmentErrorrr rrr r ValueErrorrrmodeling_utilsraropenrreadrmsgpack exceptions ExtraData startswithOSErrorUnicodeDecodeErrorr6 tree_utilr7r eval_shaperMrandomPRNGKeysetrr: _missing_keysshapelen __class____name__infor )"r$rNr" model_argsr&r%rOrPrQrRrSrTrUrVr` unused_kwargsrW model_kwargspretrained_path_with_subfolder model_fileerrrapytorch_model_filestatestate_fefparams_shape_treerequired_params shape_state missing_keysunexpected_keysr@unexpected_keyr#r#r'from_pretrainedsxT                           zFlaxModelMixin.from_pretrainedTFsave_directoryis_main_process push_to_hubcKstj|rtd|ddStj|dd|rH|dd}|dd}|dd }|d d} |d |tjjd } t | d|| d j } |} |rQ| |tj |t } t| d} t|}| |Wdn1sqwYtd| |r|j|| | ||ddSdS)a  Save a model and its configuration file to a directory so that it can be reloaded using the [`~FlaxModelMixin.from_pretrained`] class method. Arguments: save_directory (`str` or `os.PathLike`): Directory to save a model and its configuration file to. Will be created if it doesn't exist. params (`dict | FrozenDict`): A `PyTree` of model parameters. is_main_process (`bool`, *optional*, defaults to `True`): Whether the process calling this is the main process or not. Useful during distributed training and you need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions. push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. zProvided path (z#) should be a directory, not a fileNT)exist_okcommit_messageprivate create_prFrTrepo_id)rrrTwbzModel weights saved in )rTrr)rorprsrjerrormakedirsrlsplitsepr r save_configrqrrwrwriter_upload_folder)r;rr)rrr&rrrrTr model_to_saveoutput_model_filer model_bytesr#r#r'save_pretraineds:         zFlaxModelMixin.save_pretrainedr,)TF)r __module__ __qualname____doc__r config_name_automatically_saved_args_flax_internal_args classmethodr(dictrr.r"rrArErHrJr6ArrayrMrrGstrroPathLikerboolrr#r#r#r'r2sB  "'''r)-ropicklertypingrr6 jax.numpynumpyr.msgpack.exceptionsryflax.core.frozen_dictrrflax.serializationrrflax.traverse_utilrr huggingface_hubr r huggingface_hub.utilsr r rrrrrutilsrrrrrrmodeling_flax_pytorch_utilsr get_loggerrrjrr#r#r#r's