o i4@s|ddlZddlZddlZddlZddlZddlmZddlZddl m Z ddl m Z m Z mZddlmZddlZddlZddlZddlZddlZddlmZmZddlZddlZddlZddlmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&dd lm'Z'Gd d d ej(j)Z*dLd d Z+dMddZ,Gdddej-j.Z/Gdddejj0Z1ddZ2 dNdejj0fddZ3 dOdej0de4de5ej0fddZ6Gdd d ej7j8Z9dPd"d#Z:d$d%Z;d&d'Zd,d-Z?d.d/Z@d0d1ZAd2d3ZBd4d5ZCd6d7ZDd8d9ZEd:d;ZFdd?ZHGd@dAdAe%jIZJGdBdCdCe%jIZKGdDdEdEe%jIZLGdFdGdGe%jIZMGdHdIdIe$ZNdeNfdJdKZOdS)QN)trange)Image ImageDraw ImageFont)override)argsPerformanceFeature)adapters adapter_maps)BypassInjectionManager)ComfyExtensionioui) ProgressBarcs:eZdZdZddfdd Z     d ddZZS) TrainGuiderzB CFGGuider with modifications for training specific logic F offloadingcstj|i|||_dSN)super__init__r)selfrrkwargs __class__6/mnt/c/Users/fbmor/ComfyUI/comfy_extras/nodes_train.pyr!s zTrainGuider.__init__Nc Cstjj|j|j|j|j|j |jd\|_|_|_ t j |jj } |dur/tj||j| }|| }|| }|| }tjj|j| |jdz|j|j||| ||||||| d } W|jn|jwtj|j|j |`|` | S)N)force_full_load force_offload)devicedtype) latent_shapes)comfysampler_helpersprepare_sampling model_patchershapeconds model_optionsr inner_model loaded_modelstorchcuda empty_cache load_device prepare_masktosamplerscast_to_load_options model_dtypepre_run inner_samplecleanupcleanup_models) rnoise latent_imagesamplersigmas denoise_maskcallback disable_pbarseedr routputrrr outer_sample%sP       zTrainGuider.outer_sample)NNFNN)__name__ __module__ __qualname____doc__rr@ __classcell__rrrrrs rcsi}|D]C\}}ttrt||d}n,ttjr/|dus*d|kr.|}ntttfrEt |krEfdd|D}|||<q|S)N full_sizercg|]}|qSrr.0ivrr kz0make_batch_extra_option_dict..) items isinstancedictmake_batch_extra_option_dictr*Tensorsizelisttuplelen)dindiciesrGnew_dictknewvrrLrrSas   rSc Cst|drt|dst|D]\}}t||d|q|St|drit|D]>\}}t|tr>t||d|q*t|tjrK| ||<q*t|tt frht|D]\}}t||d|d|qVq*|S)N__iter__rP.) hasattr enumerateprocess_cond_listrVrPrQrRr*rTclonerW)rYprefixindexitemr\rMrrrrcps   rcc @sneZdZdddddejdddf ddZddZ dd d Zd d ZddZ ddZ ddZ   dddZ dS) TrainSamplerNrFc Cs|||_||_||_||_||_||_||_||_| |_| |_ | r%t j nd|_ | dur3|| dSd|_d|_d|_dSr)loss_fn optimizer loss_callback batch_size total_stepsgrad_accr>training_dtype real_datasetbucket_latentsr*amp GradScaler grad_scaler_init_bucket_databucket_offsetsbucket_weights num_images) rrjrkrlrmrornr>rprqrruse_grad_scalerrrrrs" zTrainSampler.__init__cCsddg|_g}|D]}||jd|j|jd|jdq|jd|_tj|tjd|_dS)z3Initialize bucket offsets and weights for sampling.rrN)rwappendr%ryr*tensorfloat32rx)rrr bucket_sizeslatrrrrvs zTrainSampler._init_bucket_dataTc s|jj|||d} |jjt|t||d} fdd|D|jd<t|||d} tj| jj |j d!|| d| dfi| } | | | }Wdn1s[wY| rz||j}|jdurv|j||S||S)NFcrHrrrIcondrrrNrOz(TrainSampler.fwd_bwd..positiverFr|T)r(model_sampling noise_scalingr* zeros_liker&rSautocastrtyperprequires_grad_rjfloatroruscalebackward)r model_wrap batch_sigmas batch_noise batch_latentrrZ extra_args dataset_sizebwdxtx0batch_extra_argsx0_predlossbwd_lossrrrfwd_bwds:   zTrainSampler.fwd_bwdcs&fddt|D}t||S)z)Generate random sigma values for a batch.cs$g|]}jjtdqS)ri)r(rpercent_to_sigmar*randrg)rJ_rrrrNs  z7TrainSampler._generate_batch_sigmas..)ranger*r~r/)rrrmrrrrr_generate_batch_sigmass z#TrainSampler._generate_batch_sigmasc st|jd}|j|}|jd} |j|t|j| } t | d|  } fdd| D} ||  |} | d| i | j }||| | j }|j|||| || ||jdd }|jre||||d |d dS) z)Execute one training step in bucket mode.rirNcsg|]}|qSrr)rJidx bucket_offsetrrrNrOz8TrainSampler._train_step_bucket_mode..samplesTr.4f)rbucket)r* multinomialrxrgrrr%rwminrmrandpermtolistr/generate_noiserrrryrl set_postfix)rrrrnoisegenr8pbar bucket_idx bucket_latent bucket_sizeactual_batch_sizerelative_indicesabsolute_indicesrrrrrrr_train_step_bucket_modes4     z$TrainSampler._train_step_bucket_modec st|d|j}tfdd|D} |d| i| j} ||t |j|| j} |j || | | ||||dd } |j rG| | | d| didS) zGExecute one training step in standard (non-bucket, non-multi-res) mode.NcrHrrrIr8rrrN rOz:TrainSampler._train_step_standard_mode..rTrrr)r*rrmrstackrr/rrrrrlrgr) rrrrrr8rrrZrrrrrrr_train_step_standard_mode s( z&TrainSampler._train_step_standard_modec Cst|d|j}d} |D]>} |j| |} |d| i| j} |jj t d } t | g| j} |j|| | | || g||dd }| |7} q| |jt|} |jdurf|j| n| |jrt|| |d| didS) zIExecute one training step in multi-resolution mode (real_dataset is set).NrrrFrrr)r*rrmrrqr/rrr(rrrrgr~rrorXrurrrlr)rrrrrr8rrrZ total_lossrf single_latentrrrrrr_train_step_multires_mode!sB   z&TrainSampler._train_step_multires_modec  Cst|j|_|jd} |d} tjt|j} t|jddt j j d} D]} t j |j| d}|jdurE||| |||| n|jdurV||| |||| | n ||| |||| | | d|jdkr|jdurv|j|j|jjD]}|dD]}|jdurq|jj|jj|j_qqz|jdur|j|j|jn|j|j | dq(tjt!|S) Nrrz Training LoRAg{Gz?)desc smoothingdisableiriparams)"rcr&rUr*r+r,rrnrr!utilsPROGRESS_BAR_ENABLED comfy_extrasnodes_custom_samplerNoise_RandomNoiser>rrrrqrrroruunscale_rk param_groupsgraddatar/rstepupdate zero_gradr)rrr:rr<r7r8r;r=rrui_pbarrrKrrparamrrrsampleEsJ                  zTrainSampler.sample)T)NNF) rArBrCr*bfloat16rrvrrrrrrrrrrrhs, % * #+rhcs4eZdZfddZddZddZddZZS) BiasDiffcst||_dSr)rrbias)rrrrrr|s  zBiasDiff.__init__cCs|j}||j|j|Sr)rr/r)rb org_dtyperrr__call__szBiasDiff.__call__cCs|j|jSr)rnelement element_sizerrrrpassive_memory_usageszBiasDiff.passive_memory_usagecCs|j|d|S)N)r)r/r)rrrrrmove_tos zBiasDiff.move_to)rArBrCrrrrrErrrrr{s  rc sd\}}td||fd}t|}t|t|fdd|D}d|t|d|f}t|ddddD]&\}} t||d|} |t| |} |j || | fgd d d | | f}qA|S) N)ii,RGBwhitecg|] }|qSrrrJlmax_lossmin_lossrrrNz#draw_loss_graph..rristartbluefillwidth) rnewrDrawrvaluesmaxintrbline) loss_mapstepsrheightimgdraw scaled_loss prev_pointrKrxyrrrdraw_loss_graphs  rmodelcCs|durg}n(t|dr/t|tjjtjjtjjfs/||t d|d|j j d|S|p2d}| D]\}}t |||d|q7|S)NforwardzFound module with forward:  ()rootr`)rarQr*nn ModuleList Sequential ModuleDictr}loggingdebugrrAnamed_children*find_all_highest_child_module_with_forward)rresultname next_namechildrrrr s  r ridepthreturnc Cs|durg}|p d}t|tjtjtjf}t|do| }|r?|d7}||kr?||td|d|d|j j d|S| D]\}}t |||||d |qC|S) a Find modules at a specific depth level for gradient checkpointing. Args: model: The model to search depth: Target depth level (1 = top-level blocks, 2 = their children, etc.) result: Accumulator for results current_depth: Current recursion depth name: Current module name for logging Returns: List of modules at the target depth NrrrizFound module at depth z: rrr`) rQrrrrrar}rrrrArfind_modules_at_depth) rrr  current_depthr  is_container has_forwardr r rrrrs "rc@s8eZdZdZedejfddZedejfddZdS) OffloadCheckpointFunctiona Gradient checkpointing that works with weight offloading. Forward: no_grad -> compute -> weights can be freed Backward: enable_grad -> recompute -> backward -> weights can be freed For single input, single output modules (Linear, Conv*). rcCsD||||_t ||WdS1swYdSr)save_for_backward forward_fnr*no_grad)ctxrrrrrrs  $z!OffloadCheckpointFunction.forwardgrad_outcCsp|j\}|j}d|_t|d}||}|||j}Wdn1s,wY~~~|dfS)NT) saved_tensorsrr* enable_graddetachrrr)rrrr x_detachedrgrad_xrrrrs  z"OffloadCheckpointFunction.backwardN) rArBrCrD staticmethodr*rTrrrrrrrs  rFcsht|dsdS|j|r t|tjtjtjtjfr fdd}n fddfdd}|_||_dS)Nrcs t|Sr)rapply)r org_forwardrrcheckpointing_fwds z patch..checkpointing_fwdcs|i|Srrrrr!rrfwdszpatch..fwdcstjjj||ddS)NF) use_reentrant)r*r checkpointr$)r%rrr#s) rarrQrLinearConv1dConv2dConv3dr")mrr#r)r%r"rpatchs    r-cCst|dr |j|_|`dSdS)Nr")rar"r)r,rrrunpatchs r.cCs g}|D] }||dq|S)zProcess latents for bucket mode training. Args: latents: list[{"samples": tensor}] where each tensor is (Bi, C, Hi, Wi) Returns: list of latent tensors rr})latentsrr latent_dictrrr_process_latents_bucket_modes r2cCsht|dkr |ddSg}|D]!}|d}|jd}|dkr,|D] }||dq!q||q|S)zProcess latents for standard (non-bucket) mode training. Args: latents: list of latent dicts or single latent dict Returns: Processed latents (tensor or list of tensors) rirrN)rXr%r})r0 latent_listlatentbs sub_latentrrr_process_latents_standard_mode$s   r7cCsFt|dkr |dSg}|D]}t|tr||q||q|S)zProcess conditioning - either single list or list of lists. Args: positive: list of conditioning Returns: Flattened conditioning list rir)rXrQrVextendr})r flat_positiverrrr_process_conditioning<s    r:c sB|r@fdd|D}t|}tdd|D}d}td|d|dt|D]\}}td |d |jq*|||fSt|tr{t}fd d|D}|D]} | | jqStd |t|d krmd}n d}t j |dd}t|}n!t|t j r| }|jd}d}ntdt|d}d}|||fS)aConvert latents to dtype and compute image counts. Args: latents: Latents (tensor, list of tensors, or bucket list) dtype: Target dtype bucket_mode: Whether bucket mode is enabled Returns: tuple: (processed_latents, num_images, multi_res) cg|]}|qSrr/rJtr|rrrN_z._prepare_latents_and_count..css|]}|jdVqdS)rN)r%r=rrr asz-_prepare_latents_and_count..Fz Bucket mode: z buckets, z total samplesz Bucket z: shape cr;rr<r=r|rrrNlr?zLatent shapes: riTr)dimzInvalid latents type: )rXsumrrrbr%rQrVsetaddr*catrTr/errorr) r0r bucket_mode num_bucketsry multi_resrKr all_shapesr4rr|r_prepare_latents_and_countRs8         rKcCsh|r|Std|dt|t|dkr|dkr||St||kr2tdt|d|d|S)a[Validate conditioning count matches image count, expand if needed. Args: positive: Conditioning list num_images: Number of images bucket_mode: Whether bucket mode is enabled Returns: Validated/expanded conditioning list Raises: ValueError: If conditioning count doesn't match image count zTotal Images: z, Total Captions: rizNumber of positive conditions (z#) does not match number of images (z).)rrrX ValueError)rryrGrrr!_validate_and_expand_conditionings rMcCsT|dkridfStd|}t|dddd}i}|r&tj|}||fS)zLoad existing LoRA weights if provided. Args: existing_lora: LoRA filename or "[None]" Returns: tuple: (existing_weights dict, existing_steps int) [None]rloras_steps_rr{) folder_pathsget_full_path_or_raisersplitr!rload_torch_file) existing_lora lora_pathexisting_stepsexisting_weightsrrr_load_existing_loras   rYcCsD|d}|jj}i}td|d|t|dkrt||dd} ||dd} d} tD]} | ||| | } | durDnq4| durMt |} | durY| |} n | j |j|dd  |} | D] \}}|||d |<qi| d |fStjtj|jj|d d }t|d }|||d <||fS)aCreate a weight adapter for a module with weight. Args: module: The module to create adapter for module_name: Name of the module existing_weights: Dict of existing LoRA weights algorithm: Algorithm name for new adapters lora_dtype: dtype for LoRA weights rank: Rank for new LoRA adapters Returns: tuple: (train_adapter, lora_params dict) .weightzCreating weight adapter for z with shape rz.alpha?z .dora_scaleN)rankalphar`Tr requires_gradz.diff)weightr%rrrXrgetr loadr to_trainr/ create_trainnamed_parameterstrainrr*r Parameterzerosr)module module_namerX algorithm lora_dtyper\keyr% lora_paramsr] dora_scaleexisting_adapter adapter_cls train_adapterr  parameterdiff diff_modulerrr_create_weight_adaptersB  rvcCsDtjtj|jj|dd}t|d}|d|i}||fS)zCreate a bias adapter for a module with bias. Args: module: The module with bias module_name: Name of the module lora_dtype: dtype for LoRA weights Returns: tuple: (bias_module, lora_params dict) Tr^z.diff_b) r*rrgrhrr%rrfr)rirjrlr bias_modulernrrr_create_bias_adapters  rxcCsi}g}|jD]U\}}t|dr^|jdur7t||||||\} } || |d} || | || t|dr^|jdur^t |||\} } || |d} || | || q ||fS)aTSetup all LoRA adapters on the model. Args: mp: Model patcher existing_weights: Dict of existing LoRA weights algorithm: Algorithm name for new adapters lora_dtype: dtype for LoRA weights rank: Rank for new LoRA adapters Returns: tuple: (lora_sd dict, all_weight_adapters list) weight_functionNrZr.bias) r named_modulesrar`rvradd_weight_wrapperr}rrx)mprXrkrlr\lora_sdall_weight_adaptersnr,adapterrrm bias_adapter bias_paramsrrr_setup_lora_adapterss(            rcCsi}g}t}|jD]{\}} t| dr| jdurXt| |||||\} } || || |d} t| t rH| | | t d| n|j | | ddt d| t| dr| jdurt| ||\} }|||d } | | | || t d | q |||fS) aSetup LoRA adapters in bypass mode. In bypass mode: - Weight adapters (lora/lokr/oft) use bypass injection (forward hook) - Bias/norm adapters (BiasDiff) still use weight wrapper (direct modification) This is useful when the base model weights are quantized and cannot be directly modified. Args: mp: Model patcher existing_weights: Dict of existing LoRA weights algorithm: Algorithm name for new adapters lora_dtype: dtype for LoRA weights rank: Rank for new LoRA adapters Returns: tuple: (lora_sd dict, all_weight_adapters list, bypass_manager) ryNrZz:[BypassMode] Added 1D weight adapter (weight wrapper) for r[)strengthz/[BypassMode] Added weight adapter (bypass) for rrzz5[BypassMode] Added bias adapter (weight wrapper) for )r rr{rar`rvrr}rQrr|rr add_adapterrrx)r}rXrkrlr\r~rbypass_managerrr,rrrmrrrrr_setup_lora_adapters_bypass&s4             rcCsd|dkr tjj||dS|dkrtjj||dS|dkr$tjj||dS|dkr0tjj||dSdS)zCreate optimizer based on name. Args: optimizer_name: Name of optimizer ("Adam", "AdamW", "SGD", "RMSprop") parameters: Parameters to optimize learning_rate: Learning rate Returns: Optimizer instance Adam)lrAdamWSGDRMSpropN)r*optimrrrr)optimizer_name parameters learning_raterrr_create_optimizer]s rcCsL|dkr tjS|dkrtjS|dkrtjS|dkr$tjSdS)zCreate loss function based on name. Args: loss_function_name: Name of loss function ("MSE", "L1", "Huber", "SmoothL1") Returns: Loss function instance MSEL1HuberSmoothL1N)r*rMSELossL1Loss HuberLoss SmoothL1Loss)loss_function_namerrr_create_loss_functionrs    rc Cstt|}tj|}|r/|ddd|ddd} |j|d| i| |||j ddS|rM|d|ddd}|j|d|i||||j ddS|j|d|i||||j ddS)aGExecute the training loop. Args: guider: The guider object train_sampler: The training sampler latents: Latent tensors num_images: Number of images seed: Random seed bucket_mode: Whether bucket mode is enabled multi_res: Whether multi-resolution mode is enabled rNrir)r>) r*r~rrrrrepeatrrr>) guider train_samplerr0ryr>rGrIr:r7 dummy_latentrrr_run_training_loops6       rc@s$eZdZeddZeddZdS) TrainLoraNodecCstjdddddtjjdddtjjdd dtjjd d dtjjd d d dddtjjdd d dddtjjddd dddtjjdddddddtjjddd d d!dtjjd"gd#d$d%d&tjjd'gd(d)d*d&tjjd+d,d,d-d.dtjjd/gd0d1d2d&tjjd3d1d4gd1d5d&tj jd6d7d8d9tjjd:t t t t d,d;d&tj jdd d d?d@dtj jdAd7dBd9tjjdCt dDdEgdEdFd&tj jdGd7dHd9tj jdId7dJd9gtdKjdLdMdNtdOjdPdQdNtjjddRdNgdSS)TNrz Train LoRAtrainingTrzThe model to train the LoRA on.tooltipr0zEThe Latents to use for training, serve as dataset/input of the model.rz.The positive conditioning to use for training.rmrii'z#The batch size to use for training.defaultrrrgrad_accumulation_stepsiz>The number of gradient accumulation steps to use for training.riz*The number of steps to train the LoRA for.rgMb@?gHz>r[z&The learning rate to use for training.)rrrrrr\zThe rank of the LoRA layers.rk)rrrrrz"The optimizer to use for training.)optionsrr loss_function)rrrrrz&The loss function to use for training.r>rlzbThe seed to use for training (used in generator for LoRA weight initialization and noise sampling)rp)bf16fp32nonerzThe dtype to use for training. 'none' preserves the model's native compute dtype instead of overriding it. For fp16 models, GradScaler is automatically enabled.rlrzThe dtype to use for lora.quantized_backwardFztWhen using training_dtype 'none' and training on quantized model, doing backward with quantized matmul when enabled.rrrkz"The algorithm to use for training.gradient_checkpointingz(Use gradient checkpointing for training.checkpoint_depthz'Depth level for gradient checkpointing.rz@Offload model weights to CPU during training to save GPU memory.rUrOrNz9The existing LoRA to append to. Set to None for new LoRA.rGzeEnable resolution bucket mode. When enabled, expects pre-bucketed latents from ResolutionBucket node. bypass_modezEnable bypass mode for training. When enabled, adapters are applied via forward hooks instead of weight modification. Useful for quantized models where weights cannot be directly modified. LORA_MODELloraz LoRA weights display_namerLOSS_MAPrz Loss historyzTotal training steps)node_idrcategoryis_experimental is_input_listinputsoutputs)r SchemaModelInputLatent ConditioningIntFloatComboBooleanrVr keysrQget_filename_listCustomOutputclsrrr define_schemas  z zTrainLoraNode.define_schemac. sJ|d}|d}|d}|d}|d}|d}| d}| d}| d} | d} | d} |d}|d}|d}|d}|d}|d}|d}|d}|tj_|rWt|}nt|}t|}|}d}t| } | dkryt| }| |n#|j }|t j krt j }| t jkrd}tjtjvrtdnt j}|dvr|nt j}t|||\}}}t|||}t d`|j dt|\}} d}!|rtdt|||| |\}"}#}!n t|||| |\}"}#t||"|} t|}$|rt |j j!|d}%t"d t#|%d ||%D] }&t$|&|d q t j%&tjj'|gd | d t j%&dgifdd}'|rIt(|$| |'||||| |||d }(nt(|$| |'||||| ||rY|nd|d }(t)||d })|)*|d}*|!dur|!+|j }*|*D]}+|+,|qxtd|!-dz6dtj_.t/|)|(||| ||Wdtj_.|*dur|*D]}+|+0|qtd|j 1D]}&t2|&qn'dtj_.|*dur|*D]}+|+0|qtd|j 1D]}&t2|&qw~(~ |"D]},|"|,3| 4|"|,<q|#D] }-|-d~-q~#t56|"|| WdS1swYdS)NrFrTaWARNING: FP16 model detected with fp16_accumulation enabled. This combination can be numerically unstable during training and may cause NaN values. Suggested fixes: 1) Set training_dtype to 'bf16', or 2) Disable fp16_accumulation (remove from --fast flags).rzUsing bypass mode for training)rz!Gradient checkpointing: patching z modules at depth rg@xD)memory_requiredrrcsd|dS)Nrr/)rrrrrlsz,TrainLoraNode.execute..loss_callback)rlrmrornr>rprrrz)rlrmrornr>rprqrzz[BypassMode] Injected z bypass hooksz![BypassMode] Ejected bypass hooks)7r!model_managementtraining_fp8_bwdr2r7r:rd node_helpersstring_to_torch_dtypeset_model_compute_dtyper get_dtyper*float16rrFp16AccumulationrfastrwarningrKrMinference_moderrYrrrrrrrdiffusion_modelinforXr-r+r,load_models_gpurhr set_condscreate_injectionsinjectget_hook_count in_trainingrejectmodulesr.r/rr NodeOutput).rrr0rrmrrrr\rkrr>rprlrrkrrrrUrGrrrr}rzrr2 latents_dtyperyrIrXrWrr~r criterionmodules_to_patchr,rlrrbypass_injections injectionrrrrrexecuteAs"                                 &zTrainLoraNode.executeNrArBrC classmethodrrrrrrrs   rc@s&eZdZeddZedddZdS)LoraModelLoadercCshtjddddtjjdddtdjd d dtjjd d d dddtjjddddgtjjdddgdS)NrzLoad LoRA ModelloadersTrz0The diffusion model the LoRA will be applied to.rrrz/The LoRA model to apply to the diffusion model.strength_modelr[gYgY@zGHow strongly to modify the diffusion model. This value can be negative.rbypassFzWhen enabled, applies LoRA in bypass mode without modifying base model weights. Useful for training and when model weights are offloaded.rzThe modified diffusion model.r)rrrrrr)r rrrrrrrrrrrrs: zLoraModelLoader.define_schemaFcCsR|dkr t|S|rtj|d||d\}}n tj|d||d\}}t|S)Nr)r rr!sdload_bypass_lora_for_modelsload_lora_for_models)rrrrr model_lorarrrrr=s     zLoraModelLoader.executeNFrrrrrrs  !rc@&eZdZeddZedddZdS)SaveLoRAcCsNtjddgddddtdjddd tjjd d d d tjjddddggdS)Nrz export lorazSave LoRA WeightsrTrrz>The LoRA model to save. Do not use the model with LoRA layers.rrezloras/ComfyUI_trained_loraz*The prefix to use for the saved LoRA file.rrzYOptional: The number of steps the LoRA has been trained for, used to name the saved file.)optionalr)rsearch_aliasesrrris_output_noderr)r rrrStringrrrrrrNs0 zSaveLoRA.define_schemaNc Csvt}t||\}}}}} |dur|d|dd} n |d|d|dd} tj|| } tj|| t S)Nr05z _.safetensorsrP) rQget_output_directoryget_save_image_pathospathjoin safetensorsr* save_filer r) rrrer output_dirfull_output_folderfilenamecounter subfolderfilename_prefixoutput_checkpointrrrrjs  zSaveLoRA.executerrrrrrrMs  rc@r) LossGraphNodec CsNtjdgdddddtdjddd tjjd d d d ggtjjtjjgd S)Nr)ztraining chartztraining visualizationz plot losszPlot Loss GraphrTrrzLoss map from training node.rr loss_graphz&Prefix for the saved loss graph image.r) rrrrrrrrhidden)r rrrrHiddenprompt extra_pnginforrrrrzs&  zLossGraphNode.define_schemaNcs|d}d\}}d}td||||fd} t| } t|t|fdd|D} t|} ||t| d|f} t| d dd d D]&\}}|t|| |}|t||}| j | ||fgd d d ||f} qI| j |df||fgdd d | j ||f|||fgdd d d}zt dd}Wn t yt }Ynw| jd|d fd|dd| j|d |dfd|dd| j|ddfd|dd| j|d|dfd|ddt| tjd}t|d}tjtj||ddS)Nr)i i(rrcrrrrrrrrNrz)LossGraphNode.execute..rrirrrrblackz arial.ttf rLoss)fontr Stepsz.2fgo@rr)r)rrrrrrrXrrbrrtruetypeIOError load_defaulttextnparrayastyperr* from_numpyr rr PreviewImage)rrrrr loss_valuesrrmarginrrrrrrKrrrr img_array img_tensorrrrrsF    zLossGraphNode.executeNNrrrrrrys  rc@s(eZdZedeeejfddZdS)TrainingExtensionrcsttttgSr)rrrrrrrr get_node_lists zTrainingExtension.get_node_listN) rArBrCrrVrr ComfyNoder-rrrrr,sr,cstSr)r,rrrrcomfy_entrypointsr/r)r^r+)riNrNr)Prrnumpyr"rr*torch.nnrtorch.utils.checkpoint tqdm.autorPILrrrtyping_extensionsrcomfy.samplersr!comfy.sampler_helperscomfy.sd comfy.utilscomfy.model_managementcomfy.cli_argsrr!comfy_extras.nodes_custom_samplerrrQrcomfy.weight_adapterr r comfy.weight_adapter.bypassr comfy_api.latestr r rrr Guider_BasicrrSrcr0SamplerrhModulerrr rrVrautogradFunctionrr-r.r2r7r:rKrMrYrvrxrrrrrr.rrrrr,r/rrrrs      E {  & %0<%7/h3,O