o i2@sddlmZddlZddlZddlmZddlZddlZddlZddl m Z ddl Z ddl m Z m Z ddlmZd1dd Zd d Zd d Zd2ddZed3ddZddZe ddZGdddZGddde jjZGddde jjZddde jfd d!Zddded"de jfd#d$Z de jfd%d&Z!dded"de jfd'd(Z"de jfd)d*Z#Gd+d,d,ej$Z%Gd-d.d.Z&ed4d/d0Z'dS)5)contextmanagerN)Path)Image)nnoptim)dataRGBcs fdd||D}||iS)z4Apply passed in transforms for HuggingFace Datasets.csg|] }|qS)convert).0imagemode transformr 5/mnt/c/Users/fbmor/ComfyUI/comfy/k_diffusion/utils.py sz+hf_datasets_augs_helper..r )examplesr image_keyrimagesr r rhf_datasets_augs_helpersrcCsV||j}|dkrtd|jd|d|dd|}|jjdkr)|S|S)zNAppends dimensions to the end of a tensor until it has target_dims dimensions.rz input has z dims but target_dims is z, which is less).Nmps)ndim ValueErrordevicetypedetachclone)x target_dimsdims_to_appendexpandedr r r append_dimss r"cCstdd|DS)z7Returns the number of trainable parameters in a module.css|]}|VqdSr)numel)r pr r r "szn_params..)sum parameters)moduler r rn_params sr)c Cst|}|jjddd|sAtj|#}t|d}t ||Wdn1s-wYWdn1s.N)modulestrain enumeraterC)modelrmodesir(r r r train_mode3s  rJcCs t|dS)zfA context manager that places a model into evaluation mode and restores the previous mode on exit.F)rJ)rGr r r eval_mode?s rKc Cst|}t|}||ksJ|D]\}}|||j|d|dqt|}t|}||ksCJ|D] \}} ||| qGdS)zIncorporates updated model parameters into an exponential moving averaged version of a model. It should be called after each optimizer step.)alphaN)dictnamed_parameterskeysitemsmul_add_ named_bufferscopy_) rGaveraged_modeldecay model_paramsaveraged_paramsnameparam model_buffersaveraged_buffersbufr r r ema_updateEs    r_c@s>eZdZdZ  dddZddZd d Zd d Zd dZdS) EMAWarmupaYImplements an EMA warmup using an inverse decay schedule. If inv_gamma=1 and power=1, implements a simple average. inv_gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), inv_gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). Args: inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. power (float): Exponential factor of EMA warmup. Default: 1. min_value (float): The minimum EMA decay rate. Default: 0. max_value (float): The maximum EMA decay rate. Default: 1. start_at (int): The epoch to start averaging at. Default: 0. last_epoch (int): The index of last epoch. Default: 0. ?rcCs(||_||_||_||_||_||_dSr) inv_gammapower min_value max_valuestart_at last_epoch)selfrcrdrerfrgrhr r r__init__hs  zEMAWarmup.__init__cCst|jS)z2Returns the state of the class as a :class:`dict`.)rN__dict__rQrir r r state_dictqszEMAWarmup.state_dictcCs|j|dS)zLoads the class's state. Args: state_dict (dict): scaler state. Should be an object returned from a call to :meth:`state_dict`. N)rkupdate)rirmr r rload_state_dictuszEMAWarmup.load_state_dictcCsLtd|j|j}dd||j|j }|dkrdSt|jt|j|S)z Gets the current EMA decay rate.rrLrb)maxrhrgrcrdminrfre)riepochvaluer r r get_value}s zEMAWarmup.get_valuecCs|jd7_dS)zUpdates the step count.rLN)rhrlr r rstepszEMAWarmup.stepN)rararbrarr) __name__ __module__ __qualname____doc__rjrmrortrur r r rr`Xs   r`c6eZdZdZ  d fdd Zdd Zd d ZZS) InverseLRaMImplements an inverse decay learning rate schedule with an optional exponential warmup. When last_epoch=-1, sets initial lr as lr. inv_gamma is the number of steps/epochs required for the learning rate to decay to (1 / 2)**power of its original value. Args: optimizer (Optimizer): Wrapped optimizer. inv_gamma (float): Inverse multiplicative factor of learning rate decay. Default: 1. power (float): Exponential factor of learning rate decay. Default: 1. warmup (float): Exponential warmup factor (0 <= warmup < 1, 0 to disable) Default: 0. min_lr (float): The minimum learning rate. Default: 0. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. rarbFcP||_||_d|krdkstdtd||_||_t|||dSNrbrLzInvalid value for warmup)rcrdrwarmupmin_lrsuperrj)ri optimizerrcrdrrrhverbose __class__r rrjzInverseLR.__init__cC|jstd|SNzTTo get the last learning rate computed by the scheduler, please use `get_last_lr()`._get_lr_called_within_stepwarningswarn_get_closed_form_lrrlr r rget_lr zInverseLR.get_lrcsDdjjddjjj fddjDS)NrLc g|] }tj|qSr rprr base_lrlr_multrirr rrz1InverseLR._get_closed_form_lr..)rrhrcrdbase_lrsrlr rrrs zInverseLR._get_closed_form_lr)rararbrbr|Frvrwrxryrjrr __classcell__r r rrr{ r{crz) ExponentialLRaEImplements an exponential learning rate schedule with an optional exponential warmup. When last_epoch=-1, sets initial lr as lr. Decays the learning rate continuously by decay (default 0.5) every num_steps steps. Args: optimizer (Optimizer): Wrapped optimizer. num_steps (float): The number of steps to decay the learning rate by decay in. decay (float): The factor by which to decay the learning rate every num_steps steps. Default: 0.5. warmup (float): Exponential warmup factor (0 <= warmup < 1, 0 to disable) Default: 0. min_lr (float): The minimum learning rate. Default: 0. last_epoch (int): The index of last epoch. Default: -1. verbose (bool): If ``True``, prints a message to stdout for each update. Default: ``False``. ?rbr|Fcr}r~) num_stepsrWrrrrrj)rirrrWrrrhrrr rrjrzExponentialLR.__init__cCrrrrlr r rrrzExponentialLR.get_lrcsBdjjdjdjjfddjDS)NrLcrr rrrr rrrz5ExponentialLR._get_closed_form_lr..)rrhrWrrrlr rrrs z!ExponentialLR._get_closed_form_lr)rrbrbr|Frr r rrrrrrbracpucCstj|||d||S)z-Draws samples from an lognormal distribution.rdtype)torchrandnexp)shapelocscalerrr r rrand_log_normalsrinfc Cstj||tjd}tj||tjd}|||}|||}tj||tjd|||} |  | | |S)zEDraws samples from an optionally truncated log-logistic distribution.r) r as_tensorfloat64logsubdivsigmoidrandlogitmuladdrto) rrrrerfrrmin_cdfmax_cdfur r rrand_log_logistics rcCs4t|}t|}tj|||d|||S)z/Draws samples from an log-uniform distribution.r)mathrrrr)rrerfrrr r rrand_log_uniforms   rc Csdt||dtj}t||dtj}tj|||d|||}t|tjd|S)zJDraws samples from a truncated v-diffusion training timestep distribution.r)ratanpirrtan) r sigma_datarerfrrrrrr r rrand_v_diffusionsrc Cs`tj|||d}tj|||d}|| |}|||} |||} t|| k|| S)z2Draws samples from a split lognormal distribution.r)rrabsrwherer) rrscale_1scale_2rrnrn_leftn_rightratior r rrand_split_log_normals   rcsBeZdZdZhdZd fdd ZddZdd Zd d ZZ S) FolderOfImageszURecursively finds all images in a directory. It does not support classes/targets.> .bmp.jpg.pgm.png.ppm.tif.jpeg.tiff.webpNcsNtt|_|durtn|_tfddjdD_ dS)Nc3s$|] }|jjvr|VqdSr)suffixlowerIMG_EXTENSIONS)r r<rlr rr% s"z*FolderOfImages.__init__..*) rrjrrootrIdentityrsortedrglobpaths)rirrrrlrrjs  $zFolderOfImages.__init__cCsd|jdt|dS)NzFolderOfImages(root="z", len: ))rlenrlr r r__repr__zFolderOfImages.__repr__cCs t|jSr)rrrlr r r__len__s zFolderOfImages.__len__cCsT|j|}t|d}t|d}Wdn1swY||}|fS)Nr-r)rr4rr r)rikeyr<r@r r r r __getitem__s   zFolderOfImages.__getitem__r) rvrwrxryrrjrrrrr r rrrsrc@seZdZddZddZdS) CSVLoggercCsJt||_||_|jrt|jd|_dSt|jd|_|j|jdS)Naw)rfilenamecolumnsr0r4filewrite)rirrr r rrjs  zCSVLogger.__init__cGst|d|jdddS)N,T)seprflush)printr)riargsr r rr&rzCSVLogger.writeN)rvrwrxrjrr r r rrs rccstjjj}tjjjj}z.|dur|tjj_|dur |tjjj_dVW|dur-|tjj_|dur9|tjjj_dSdS|durD|tjj_|durN|tjjj_w)zGA context manager that sets whether TF32 is allowed on cuDNN or matmul.N)rbackendscudnn allow_tf32cudamatmul)rr cudnn_old matmul_oldr r r tf32_mode*s$      r)rr)T)NN)( contextlibrr7rpathlibrr5r1rPILrrrr torch.utilsrrr"r)rBrJrKno_gradr_r` lr_scheduler _LRSchedulerr{rfloat32rfloatrrrrDatasetrrrr r r rs@         0))