o TiA@sddlZddlZddlZddlmZmZddlZddlmZddl m Z ddl m Z eej dr6ej jdusJzddlZdZWn eyIdZYnwdZd d lmZd d lmZd d lmZd d lmZd dlmZmZmZd dlmZmZd dl m!Z!d dl"m#Z#d dl$m%Z%d dl&m'Z'd dl(m)Z)d dl*m+Z+m,Z,d dl-m.Z.d dl/m0Z0m1Z1d dlm2Z2m3Z3m4Z4d dl5m6Z6m7Z7m8Z8d dl9m:Z:d dl;mm?Z?d dl@mAZAd dlBm Z mCZCmDZDdd ZEe ZFeEeF\ZGZHZIeCZJeDZKdaLddddddeddddddf d!ejMjNd"eeeefd#eejMjNd$eej5jOjPd%eee efd&eQd'eeRfd(d)ZSd*d+ZTd,d-ZUd.d/ZVd4d0d1ZWd4d2d3ZXdS)5N)OptionalUnion) Optimizer) _LRScheduler)versionhipTF)ops) module_inject)get_accelerator)TORCH_DISTRIBUTED_DEFAULT_PORT)DeepSpeedEngineDeepSpeedOptimizerCallableDeepSpeedSchedulerCallable)ADAM_OPTIMIZERLAMB_OPTIMIZER)DeepSpeedHybridEngine)PipelineEngine)InferenceEngine)DeepSpeedInferenceConfig)add_tuning_arguments)DeepSpeedConfigDeepSpeedConfigError) checkpointing)DeepSpeedTransformerLayerDeepSpeedTransformerConfig)replace_transformer_layerrevert_transformer_layerset_autotp_mode)log_distOnDevicelogger)init_distributed)zerodomino)is_compile_supported)PipelineModule)rgit_hash git_branchcCst|}|j|j|jfS)zHParse a version string and extract the major, minor, and patch versions.) pkg_versionparsemajorminormicro) version_strverr0F/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed/__init__.py_parse_version5s r2model optimizermodel_parameters training_data lr_schedulerdistributed_portdist_init_requiredc Cstdtttdgdtj|dusJdddlm a t } t j | ||d| dur6| dur6| } d}| rItd| t | d }n | durid | vrid | vritd | t | d | d fd }t|d r|jdurtdt|dr|jdusJd|j|_d|_t|dr|jdur| dusJd|j} | dusJdt|tst| ||d}|jjrt||||||||| | |d }n1t||||||||| | ||d }n |dusJd|}t| |}t||||||||| | |d }tj||j|j |j!g}t"|S)ar Initialize the DeepSpeed Engine. Arguments: args: an object containing local_rank and deepspeed_config fields. This is optional if `config` is passed. model: Required: nn.module class before apply any wrappers optimizer: Optional: a user defined Optimizer or Callable that returns an Optimizer object. This overrides any optimizer definition in the DeepSpeed json config. model_parameters: Optional: An iterable of torch.Tensors or dicts. Specifies what Tensors should be optimized. training_data: Optional: Dataset of type torch.utils.data.Dataset lr_scheduler: Optional: Learning Rate Scheduler Object or a Callable that takes an Optimizer and returns a Scheduler object. The scheduler object should define a get_lr(), step(), state_dict(), and load_state_dict() methods distributed_port: Optional: Master node (rank 0)'s free port that needs to be used for communication during distributed training mpu: Optional: A model parallelism unit object that implements get_{model,data}_parallel_{rank,group,world_size}() dist_init_required: Optional: None will auto-initialize torch distributed if needed, otherwise the user can force it to be initialized or not via boolean. collate_fn: Optional: Merges a list of samples to form a mini-batch of Tensor(s). Used when using batched loading from a map-style dataset. config: Optional: Instead of requiring args.deepspeed_config you can pass your deepspeed config as an argument instead, as a path or a dictionary. config_params: Optional: Same as `config`, kept for backwards compatibility. Returns: A tuple of ``engine``, ``optimizer``, ``training_dataloader``, ``lr_scheduler`` * ``engine``: DeepSpeed runtime engine which wraps the client model for distributed training. * ``optimizer``: Wrapped optimizer if a user defined ``optimizer`` is supplied, or if optimizer is specified in json config else ``None``. * ``training_dataloader``: DeepSpeed dataloader if ``training_data`` was supplied, otherwise ``None``. * ``lr_scheduler``: Wrapped lr scheduler if user ``lr_scheduler`` is passed, or if ``lr_scheduler`` specified in JSON configuration. Otherwise ``None``. 6DeepSpeed info: version={}, git-hash={}, git-branch={}rranksNz%deepspeed.initialize requires a model)comm) dist_backendr8r9z&mesh_param to Initialize mesh device: ) data_parallelsequence_parallelsequence_parallel_sizedata_parallel_sizez"config to Initialize mesh device: deepscale_configzY************ --deepscale_config is deprecated, please use --deepspeed_config ************deepspeed_configzSNot sure how to proceed, we were given both a deepscale_config and deepspeed_configz|Not sure how to proceed, we were given deepspeed configs in the deepspeed arguments and deepspeed.initialize() function callzCDeepSpeed requires --deepspeed_config to specify configuration file) mesh_device) argsr3r4r5r6r7mpur9 collate_fnconfig config_class) rFr3r4r5r6r7rGr9rHrIrErJz*mpu must be None with pipeline parallelism)#rformat __version__ __git_hash____git_branch__r#partition_parametersshutdown_init_context deepspeedr=distr communication_backend_namer"r!infoinitialize_mesh_devicehasattrrCwarningrD isinstancer&r hybrid_engineenabledrr rGrrestore_init_contextr4training_dataloaderr7tuple)rFr3r4r5r6r7r8rGr9rHrI mesh_param config_paramsr>rErJengine return_itemsr0r0r1 initializeEs ?          rbcCsX|dd}|jddddd|jdd td d |jd ddd d|jdd tdd |S)aHelper (internal) function to update an argument parser with an argument group of the core DeepSpeed arguments. The core set of DeepSpeed arguments include the following: 1) --deepspeed: boolean flag to enable DeepSpeed 2) --deepspeed_config : path of a json configuration file to configure DeepSpeed runtime. This is a helper function to the public add_config_arguments() Arguments: parser: argument parser Return: parser: Updated Parser DeepSpeedzDeepSpeed configurationsz --deepspeedF store_truezLEnable DeepSpeed (helper flag for user code, no impact on DeepSpeed backend))defaultactionhelpz--deepspeed_configNz"DeepSpeed json configuration file.)retypergz --deepscalezWDeprecated enable DeepSpeed (helper flag for user code, no impact on DeepSpeed backend)z--deepscale_configz-Deprecated DeepSpeed json configuration file.)add_argument_group add_argumentstr)parsergroupr0r0r1_add_core_argumentss$ rncCs t|}|S)aUpdate the argument parser to enabling parsing of DeepSpeed command line arguments. The set of DeepSpeed arguments include the following: 1) --deepspeed: boolean flag to enable DeepSpeed 2) --deepspeed_config : path of a json configuration file to configure DeepSpeed runtime. Arguments: parser: argument parser Return: parser: Updated Parser )rn)rlr0r0r1add_config_arguments s rocCs tS)zL Return a default DeepSpeed inference configuration dictionary. )rdictr0r0r0r1default_inference_configs rqc Kstdtttdgd|duri}t|tr2t|d }t |}Wdn1s,wYnt|t r:|}n t dt |t ||}|D]}||||krkt d|d||d ||qP||td i|}t||d }|S) aInitialize the DeepSpeed InferenceEngine. Description: all four cases are valid and supported in DS init_inference() API. # Case 1: user provides no config and no kwargs. Default config will be used. .. code-block:: python generator.model = deepspeed.init_inference(generator.model) string = generator("DeepSpeed is") print(string) # Case 2: user provides a config and no kwargs. User supplied config will be used. .. code-block:: python generator.model = deepspeed.init_inference(generator.model, config=config) string = generator("DeepSpeed is") print(string) # Case 3: user provides no config and uses keyword arguments (kwargs) only. .. code-block:: python generator.model = deepspeed.init_inference(generator.model, tensor_parallel={"tp_size": world_size}, dtype=torch.half, replace_with_kernel_inject=True) string = generator("DeepSpeed is") print(string) # Case 4: user provides config and keyword arguments (kwargs). Both config and kwargs are merged and kwargs take precedence. .. code-block:: python generator.model = deepspeed.init_inference(generator.model, config={"dtype": torch.half}, replace_with_kernel_inject=True) string = generator("DeepSpeed is") print(string) Arguments: model: Required: original nn.module object without any wrappers config: Optional: instead of arguments, you can pass in a DS inference config dict or path to JSON file Returns: A deepspeed.InferenceEngine wrapped model. r:rr;Nrz5'config' argument expected string or dictionary, got zConflicting argument 'z' in 'config':z and kwargs:)rIr0)rrKrLrMrNrXrkopenjsonloadrp ValueErrorrhsetkeys intersectionupdaterr) r3rIkwargsf config_dict overlap_keyskeyds_inference_configr`r0r0r1init_inference#s0 0    "  rcKsNt|dr tddStddddlm}||||dj}t|dd|S) aN Initialize the model for tensor parallelism. Args: model (torch.nn.Module): The model to be initialized. tp_size (int): The tensor parallelism size. dtype (torch.dtype): The data type to be used for the model. Returns: torch.nn.Module: The initialized model with tensor parallelism. ds_autotp_parsedzQds_autotp_parsed' attribute already exists in the model, re-entry is not allowed.NT)trainingr)TpTrainingManager)r3tp_sizedtype)rVr!rWr!deepspeed.runtime.tensor_parallelrmodulesetattr)r3rrrIr{rr0r0r1 tp_model_initqs    r)N)Ysystypesrttypingrrtorch torch.optimrtorch.optim.lr_schedulerr packagingrr)rVrtriton HAS_TRITON ImportErrorr r acceleratorr constantsr runtime.enginer rrrrruntime.hybrid_enginerruntime.pipe.enginerinference.enginerinference.configrruntime.lr_schedulesrruntime.configrr runtime.activation_checkpointingrops.transformerrrrrrutilsrr r! comm.commr"runtimer#r$runtime.compilerr%piper&git_version_infor'r(r2rL__version_major____version_minor____version_patch__rMrNrRnnModuledataDatasetintboolrbrnrorqrrr0r0r0r1s                    %# N