o ߥim@sdZddlZddlZddlZddlZddlZddlmZddZddZ d d Z d d Z d dZ ddZ ddZddZddZdS)zargparser configurationN) get_hostnamecCs6|dd}|jdddd|jdddd|jd dd d|jd td d d|jdtddd|jdtddd|jdtddd|jdtddd|jdtddd|jdtd dd|jd td d!d|jd"td#d$d|jd%td&d'd|jd(dd)d|jd*td+d,d|jd-dd.d|jd/dd0d|S)1zModel argumentsmodelzmodel configurationz--transformer-xl store_truezuse transformer-xl for trainingactionhelpz--pretrained-bertzuse a pretrained bert-large-uncased model insteadof initializing from scratch. See --tokenizer-model-type to specify which pretrained BERT model to usez--encoder-decoderz0use the encoder-decoder architecture for blocklmz--attention-dropout皙?z)dropout probability for attention weightstypedefaultrz--num-attention-headsz"num of transformer attention headsz --hidden-sizeiztansformer hidden sizez--intermediate-sizeNzMtransformer embedding dimension for FFNset to 4*`--hidden-size` if it is Nonez --num-layersznum decoder layersz--layernorm-epsilongh㈵>zlayer norm epsilonz--hidden-dropoutz0dropout probability for hidden state transformerz--output-dropoutz%dropout probability for pooled outputz--max-position-embeddingsz,maximum number of position embeddings to usez --vocab-sizeiznvocab size to use for non-character-level tokenization. This value will only be used when creating a tokenizerz --deep-initzinitialize bert model similar to gpt2 model.scales initialization of projection layers by a factor of 1/sqrt(2N). Necessary to train bert models larger than BERT-Large.z--make-vocab-size-divisible-byzcPad the vocab size to be divisible by this value.This is added for computational efficieny reasons.z--cpu-optimizerzRun optimizer on CPUz--cpu_torch_adamz#Use Torch Adam as optimizer on CPU.add_argument_group add_argumentfloatintparsergrouprX/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/mglm/arguments.pyadd_model_config_argss rcCs|dd}|jdddd|jdddd|jd dd d|jd dd d|jd ddd|jdtddd|jdtddd|jdtddd|jdtddd|jdtdd|S)zMixed precision arguments.fp16zfp16 configurationsz--fp16rzRun model in fp16 moderz--fp32-embeddingzembedding in fp32z--fp32-layernormzlayer norm in fp32z--fp32-tokentypeszembedding token types in fp32z--fp32-allreducezall-reduce in fp32z --hysteresisz#hysteresis for dynamic loss scalingr z --loss-scaleNzsStatic loss scaling, positive power of 2 values can improve fp16 convergence. If None, dynamicloss scaling is used.z--loss-scale-windowz.Window over which to raise/lower dynamic scalez --min-scalerz)Minimum loss scale for dynamic loss scalez--attention-scale?r r )rrrrrrrradd_fp16_config_argstsX r!cCs|dd}|jdtddd|jdtdd d|jd td d d|jd td dd|jdddd|jdtd dd|jdddd|jdtddd|jdtddd|jdtddd|jd td!d"|jd#td$d%d|jd&td'd(d|jd)td*d+d|jd,dd-d|jd.dd/d|jd0tdd1d|jd2td3gd4d5d6|jd7td8d"|jd9td:d;d|jddd?d|jd@tddAd|jdBddC|jdDtd dEd|jdFtdGdHd|jdIddJd|jdKddLd|jdMtddNd|jdOddPd|jdQddRd|jdSddTd|jdUddVd|jdWddXd|jdYddZd|jd[d\d]d\d^gd_|jd`dagdbdcdd|jdetddfd|jdgddhd|jdiddjd|jdktdld"|jdmtdld"|jdntdld"|jdotd!d"|jdptdqd"|jdrtdsd"|jdttd!d"|jdutd!d"|jdvddwd|jdxddyd|jdzdd{d|jd|dd}d|jd~td!d"|jdtd!d"|jdddd|S)zTraining arguments.trainztraining configurationsz--experiment-namezgpt-345Mz.The experiment name for summary and checkpointr z --batch-sizezData Loader batch sizez--gradient-accumulation-stepsrz--weight-decayg{Gz?z.weight decay coefficient for L2 regularizationz--checkpoint-activationsrzLcheckpoint activation to allow for training with larger models and sequencesrz--checkpoint-num-layersz/chunk size (number of layers) for checkpointingz$--deepspeed-activation-checkpointingz,uses activation checkpointing from deepspeedz--epochsNz>Number of finetunning epochs. Zero results in evaluation only.z --clip-gradrzgradient clippingz --train-itersrz:total number of iterations to train over all training runsz--label-smoothingr z--log-intervaldzreport intervalz --summary-dirz"The directory to store the summaryz--seediz random seedz--reset-position-idsz0Reset posistion ids after end-of-document token.z--reset-attention-maskz7Reset self attention maske after end-of-document token.z--lr-decay-iterszUnumber of iterations to decay LR over, If None defaults to `--train-iters`*`--epochs`z--lr-decay-stylelinear)constantr'cosine exponentialzlearning rate decay functionr r choicesrz--lr-decay-ratior z--lrg-C6?zinitial learning ratez--warmupzNpercentage of data to warmup on (.01 = 1% of all training iters). Default 0.01z--switch-linearz'Switch to linear decay for cosine decayz--savez(Output directory to save checkpoints to.z--new-save-directoryrz --save-epochznumber of epochs between savesz--save-intervaliz"number of iterations between savesz--no-save-optimzDo not save current optimizer.z --no-save-rngzDo not save current rng state.z--loadz2Path to a directory containing a model checkpoint.z--no-load-optimz.Do not load optimizer when loading checkpoint.z --no-load-rngz.Do not load rng state when loading checkpoint.z--no-load-lr-schedulerz1Do not load lr scheduler when loading checkpoint.z--no-deepspeed-loadz)Not use deepspeed when loading checkpointz --finetunezLoad model for finetuning. Do not load optimizer or rng state from checkpoint and set iteration to 0. Assumed when loading a release checkpoint.z--resume-dataloaderzResume the dataloader when resuming training. Does not apply to tfrecords dataloader, try resumingwith a different seed in this case.z--distributed-backendncclzBwhich backend to use for distributed training. One of [gloo, nccl]gloo)r rr,z --DDP-impltorch)localr0nonez4which DistributedDataParallel implementation to use.)r r,rz --local_rankz+local rank passed from distributed launcherz --block-lmz$whether use the BlockLM pre-trainingz --masked-lmz whether to use the mlm objectivez --bert-probg?z--gpt-infill-probz--gpt-min-ratioz--gap-sentence-probz--gap-sentence-ratiog333333?z--avg-block-lengthz--short-seq-probz--single-span-probz --task-maskz3Use different mask for generation and blank fillingz--no-shuffle-blockz-not shuffle the blocks when filling the blankz--no-block-positionz9Use (rough) absolute positions instead of block positionsz--sentinel-tokenz:Use sentinel (mask) tokens to replace 2d position encodingz--block-mask-probz--context-mask-ratioz--random-positionz>Use random start position to cover all the position embeddingsrrstrrrrrrradd_training_argss r6cCs|dd}|jdtddd|jdtdd d|jd td d d|jd tddd|jdtddd|jdtddd|jdtdd|S)zEvaluation arguments. validationzvalidation configurationsz--eval-batch-sizeNzIData Loader batch size for evaluation datasets.Defaults to `--batch-size`r z --eval-itersr%z=number of iterations to run for evaluationvalidation/test forz--eval-intervalrz5interval between running evaluation on validation setz --eval-epochrz2epoch between running evaluation on validation setz--eval-seq-lengthzMMaximum sequence length to process for evaluation. Defaults to `--seq-length`z--eval-max-preds-per-seqziMaximum number of predictions to use for evaluation. Defaults to math.ceil(`--eval-seq-length`*.15/10)*10z--overlapping-eval r )rrrrrrradd_evaluation_argsssRr9cCs|dd}|jdtdd|jdtdd|jdtd d|jd td d|jd td d|jdtdd|jdtd d|jdtd d|jddd|jdtdd|S)zText generate arguments.zText generationconfigurationsz --temperaturerr z--top_pr$z--top_krz--out-seq-lengthz --num-beamsrz--length-penaltyz--no-repeat-ngram-sizez--min-tgt-lengthz --select-topkrr-z--blank-maskratior rrrrradd_text_generate_argss r<cCsx|dd}|jdtddd|jddd d |jd dd |jd dddd|jddddd|jddddd|jdtddd|jdtddd|jdddd|jd d!d"d|jd#dd$d|jd%d&d'd|jd(dd)d |jd*dd |jd+tdd,d|jd-dd.d |jd/dd0d |jd1td2d3d|jd4tdd5d|jd6td7d8d|jd9td:gd;ddd |jd?dtd@dA|jdBddCd |jdDtdEdFd|jdGtdHdId|jdJtddKd|jdLtdMdN|jdOddPd |jdQtddRd|jdStddTd|jdUtddVd|jdWdddXd|jdYtdMdZd|jd[tddN|jd\tddN|S)]z Train/valid/test data arguments.datazdata configurationsz--model-parallel-sizerzsize of the model parallel.r z --shufflerzIShuffle data. Shuffling is deterministic based on seed and current epoch.rz--filter-englishr-z --train-data+Nz=Whitespace separated filenames or corpora names for training.)nargsr rz --valid-data*zFilename for validation data.z --test-datazFilename for testingz --data-dirz#The data path to all the data filesz--input-data-sizes-filez sizes.txtz,the filename containing all the shards sizesz--delim,z&delimiter used to parse csv data files)r rz --text-keysentencez(key to use to extract text from json/csvz--eval-text-keyzd?|jd@ddAdB|jdCtdd|jdDtdd|jdEddAdB|S)FNfinetunezfinetune configurationsz--taskz Task name.)r rz--load-pretrainedzLoad pretrained model)r rr z --pool-token)startpadclsz-The token to pool the sequence representationrL)r r,rr z --cloze-evalrz"Evaluation dataset with cloze taskrz --multi-tokenz$Use multi token for cloze evaluationz--segment-lengthrz/The maximum segment length for cloze evaluationr z --loss-func) cross_entropyhinge generativemixrM)r r,r z--block-lm-ratior$r z--adapetz.Use the decoupled cross entropy loss in AdaPETz --pattern-idz --fast-decodezRFast decode for multi-token cloze. Can only be used without checkpoint activation.z--few-supergluer-z --eval-validz!Whether evaluate on the valid setz--validation-metricz--unidirectionalz$Use the left to right language modelz--src-seq-lengthz--tgt-seq-lengthz --adam-beta1g?z --adam-beta2g+?z --adam-epsg:0yE>z --optimizeradam adafactorz--wsc-negativez --overwritez--no-validationz--continuous-promptzUse continuous prompt for PETz--num-prompt-tokensz --prompt-funclstm)rSmlpr2)r r,z--freeze-transformerF)rr z--tune-prefix-layersz--prefix-promptz --prompt-initr4rrrradd_finetune_config_args_s   rUc CsNtjdd}t|}t|}t|}t|}t|}t|}t|}t |}|j gd}|j s7|j s7tdtj|_ttdd|_ttdd|_t|d r\|jr\t|n2td rttd }ttd }ttd d}ttd d}||_||||_|||_t|j|j|_|jdkrtd|j|jd|_|jdurd|_|jdkrtd|j sd|_!d|_"d|_#t|dr%|j r%|j$dur%t%|j$dd }t&'|}Wdn1swYd|vr|d|_(d|vr|d|_)nd|_)d|vr%|d*di}|*d|j+|_+|*d|j,|_,|S)zParse all the args.zPyTorch BERT Model) description)argsz#WARNING: No training data specifiedRANK0 WORLD_SIZE1 deepspeed_mpiOMPI_COMM_WORLD_LOCAL_RANKOMPI_COMM_WORLD_LOCAL_SIZESLURM_JOB_NUM_NODES SLURM_NODEIDrz1using world size: {} and model-parallel size: {} FNTz > using dynamic loss scaling deepspeedzutf-8)encodingtrain_micro_batch_size_per_gpugradient_accumulation_stepsr optimizerparamslr weight_decay)-argparseArgumentParserrr!r6r9r<rHrUraadd_config_arguments parse_args train_datadata_dirprintr0cuda is_availablerosgetenvrank world_sizehasattrr\mpi_define_env local_rankminmodel_parallel_sizeformatdynamic_loss_scale loss_scalerfp32_embeddingfp32_tokentypesfp32_layernormdeepspeed_configopenjsonload batch_sizerdgetrgrh) rrWrx local_size num_nodesnodeidfileroptimizer_params_configrrrget_argss                rc sddlm}|j}|}|}d}|dkrt}|j|dd}||}t fdd|d|D}t |t j d<t |t j d<||_ ||_||_|t j d<d t j d <td t j d|j t j dt j dt j d dS) Nr)MPI)rootcsg|]}|kqSrr).0i proc_namerr sz"mpi_define_env..rXrZ MASTER_ADDR29500 MASTER_PORTzfDiscovered MPI settings of world_rank={}, local_rank={}, world_size={}, master_addr={}, master_port={})mpi4pyr COMM_WORLDGet_rankGet_sizerbcastGet_processor_name allgathersumr5rrenvironrxrurtror{)rWrcommrtru master_addr all_procsrxrrrrws6   rw)__doc__rirrrarr0utilsrrr!r6r9r<rHrUrrwrrrrs$  Z+U..P N