can not train models

[MachineTrooper]

It appears Kohya won't let me train using anything (textual inversion, Dreambooth, etc.) but the one I most would like to use is Lora. I've followed all the directions from tutorials I've found on installing and tweaking the settings in Kohya, but all Kohya SS will do is create an empty folder that says "sample."

Thanks in advance for any guidance you can give me.

[0ka9dkad]

I have the same problem as you.

#870

Until now nobody did give me a answer.

[0ka9dkad]

I think i did "solve the problem". I can send you my .json file and you see for yourself, also when you put your directory location you mustn't put in the end \ for example:

Don't put

E:\Projectos meus\AI Art\Loras\onpu1\image
E:\Projectos meus\AI Art\Loras\onpu1\model
E:\Projectos meus\AI Art\Loras\onpu1\log\

Instead try this

E:\Projectos meus\AI Art\Loras\onpu1\image
E:\Projectos meus\AI Art\Loras\onpu1\model
E:\Projectos meus\AI Art\Loras\onpu1\log

For me this did the trick... So stupid the bug but it did work for me...

[MachineTrooper]

Even try to train some models i did some time and none of them work. I don't know what to do... What gpu are you using?

NVidia GeForce RTX 3060.

[MachineTrooper]

I think i did "solve the problem". I can send you my .json file and you see for yourself, also when you put your directory location you mustn't put in the end \ for example:

Don't put

E:\Projectos meus\AI Art\Loras\onpu1\image E:\Projectos meus\AI Art\Loras\onpu1\model E:\Projectos meus\AI Art\Loras\onpu1\log\

Instead try this

E:\Projectos meus\AI Art\Loras\onpu1\image E:\Projectos meus\AI Art\Loras\onpu1\model E:\Projectos meus\AI Art\Loras\onpu1\log

For me this did the trick... So stupid the bug but it did work for me...

Wow, that would be crazy. So just the backslash at the end of the log address?

[MachineTrooper]

I think i did "solve the problem". I can send you my .json file and you see for yourself, also when you put your directory location you mustn't put in the end \ for example:

Don't put

E:\Projectos meus\AI Art\Loras\onpu1\image E:\Projectos meus\AI Art\Loras\onpu1\model E:\Projectos meus\AI Art\Loras\onpu1\log\

Instead try this

E:\Projectos meus\AI Art\Loras\onpu1\image E:\Projectos meus\AI Art\Loras\onpu1\model E:\Projectos meus\AI Art\Loras\onpu1\log

For me this did the trick... So stupid the bug but it did work for me...

Nope. Same exact result. No change.

[0ka9dkad]

for me did the trick, also i use a less power gpu than you. anyway, if you how to resolve your problem let me know!

[sgtsixpack]

I have the same problem (says returned non-zero exit status 1.) Any help here?

Folder 100_LoraJanaDefi1st: 110 images found Folder 100_LoraJanaDefi1st: 11000 steps Total steps: 11000 Train batch size: 1 Gradient accumulation steps: 1.0 Epoch: 1 Regulatization factor: 1 max_train_steps (11000 / 1 / 1.0 * 1 * 1) = 11000 stop_text_encoder_training = 0 lr_warmup_steps = 1100 accelerate launch --num_cpu_threads_per_process=2 "train_network.py" --enable_bucket --pretrained_model_name_or_path="J:/AI training/Stable diffusion/stable-diffusion-webui-directml/models/Stable-diffusion/hardblend_.safetensors" --train_data_dir="J:/AI training/Stable diffusion/stable-diffusion-webui-directml/SgtSixpack Training/LoraJanaDefi1st/image1st" --resolution=512,512 --output_dir="J:/AI training/Stable diffusion/stable-diffusion-webui-directml/SgtSixpack Training/LoraJanaDefi1st/model1st" --logging_dir="J:/AI training/Stable diffusion/stable-diffusion-webui-directml/SgtSixpack Training/LoraJanaDefi1st/log1st" --network_alpha="1" --training_comment="hardblend_.safetensors training" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=5e-05 --unet_lr=0.0001 --network_dim=8 --output_name="JanaDefiNigel_HarrLora" --lr_scheduler_num_cycles="1" --learning_rate="0.0001" --lr_scheduler="cosine" --lr_warmup_steps="1100" --train_batch_size="1" --max_train_steps="11000" --save_every_n_epochs="1" --mixed_precision="fp16" --save_precision="fp16" --cache_latents --optimizer_type="AdamW8bit" --max_data_loader_n_workers="0" --bucket_reso_steps=64 --xformers --bucket_no_upscale 2023-06-01 18:00:03.077447: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found 2023-06-01 18:00:03.077524: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2023-06-01 18:00:07.046971: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found 2023-06-01 18:00:07.047048: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for: PyTorch 2.0.0+cu118 with CUDA 1108 (you have 2.0.1+cpu) Python 3.10.11 (you have 3.10.6) Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers) Memory-efficient attention, SwiGLU, sparse and more won't be available. Set XFORMERS_MORE_DETAILS=1 for more details J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: 'Could not find module 'J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\Lib\site-packages\torchvision\image.pyd' (or one of its dependencies). Try using the full path with constructor syntax.'If you don't plan on using image functionality from torchvision.io, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have libjpegorlibpnginstalled before buildingtorchvision` from source?
warn(
prepare tokenizer
Use DreamBooth method.
prepare images.
found directory J:\AI training\Stable diffusion\stable-diffusion-webui-directml\SgtSixpack Training\LoraJanaDefi1st\image1st\100_LoraJanaDefi1st contains 110 image files
11000 train images with repeating.
0 reg images.
no regularization images / 正則化画像が見つかりませんでした
[Dataset 0]
batch_size: 1
resolution: (512, 512)
enable_bucket: True
min_bucket_reso: 256
max_bucket_reso: 1024
bucket_reso_steps: 64
bucket_no_upscale: True

[Subset 0 of Dataset 0]
image_dir: "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\SgtSixpack Training\LoraJanaDefi1st\image1st\100_LoraJanaDefi1st"
image_count: 110
num_repeats: 100
shuffle_caption: False
keep_tokens: 0
caption_dropout_rate: 0.0
caption_dropout_every_n_epoches: 0
caption_tag_dropout_rate: 0.0
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: False
token_warmup_min: 1,
token_warmup_step: 0,
is_reg: False
class_tokens: LoraJanaDefi1st
caption_extension: .caption

[Dataset 0]
loading image sizes.
100%|██████████████████████████████████████████████████████████████████████████████| 110/110 [00:00<00:00, 2928.93it/s]
make buckets
min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is set, because bucket reso is defined by image size automatically / bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計算されるため、min_bucket_resoとmax_bucket_resoは無視されます
number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）
bucket 0: resolution (320, 512), count: 100
bucket 1: resolution (512, 512), count: 10900
mean ar error (without repeats): 0.0004794034090909091
prepare accelerator
J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py:249: FutureWarning: logging_dir is deprecated and will be removed in version 0.18.0 of 🤗 Accelerate. Use project_dir instead.
warnings.warn(
Using accelerator 0.15.0 or above.
loading model for process 0/1
load StableDiffusion checkpoint: J:/AI training/Stable diffusion/stable-diffusion-webui-directml/models/Stable-diffusion/hardblend_.safetensors
J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\safetensors\torch.py:98: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
with safe_open(filename, framework="pt", device=device) as f:
loading u-net:
loading vae:
loading text encoder:
CrossAttention.forward has been replaced to enable xformers.
[Dataset 0]
caching latents.
0%| | 0/110 [00:00<?, ?it/s]
Traceback (most recent call last):
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\train_network.py", line 783, in
train(args)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\train_network.py", line 157, in train
train_dataset_group.cache_latents(vae, args.vae_batch_size, args.cache_latents_to_disk, accelerator.is_main_process) File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\library\train_util.py", line 1399, in cache_latents
dataset.cache_latents(vae, vae_batch_size, cache_to_disk, is_main_process)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\library\train_util.py", line 812, in cache_latents
latents = vae.encode(img_tensors).latent_dist.sample().to("cpu")
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\diffusers\models\vae.py", line 566, in encode
h = self.encoder(x)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\diffusers\models\vae.py", line 130, in forward
sample = self.conv_in(sample)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: "slow_conv2d_cpu" not implemented for 'Half'
Traceback (most recent call last):
File "J:\AI training\Python310\lib\runpy.py", line 196, in run_module_as_main
return run_code(code, main_globals, None,
File "J:\AI training\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\Scripts\accelerate.exe_main.py", line 7, in
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main
args.func(args)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 923, in launch_command
simple_launcher(args)
File "J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 579, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['J:\AI training\Stable diffusion\stable-diffusion-webui-directml\kohya_ss\venv\Scripts\python.exe', 'train_network.py', '--enable_bucket', '--pretrained_model_name_or_path=J:/AI training/Stable diffusion/stable-diffusion-webui-directml/models/Stable-diffusion/hardblend.safetensors', '--train_data_dir=J:/AI training/Stable diffusion/stable-diffusion-webui-directml/SgtSixpack Training/LoraJanaDefi1st/image1st', '--resolution=512,512', '--output_dir=J:/AI training/Stable diffusion/stable-diffusion-webui-directml/SgtSixpack Training/LoraJanaDefi1st/model1st', '--logging_dir=J:/AI training/Stable diffusion/stable-diffusion-webui-directml/SgtSixpack Training/LoraJanaDefi1st/log1st', '--network_alpha=1', '--training_comment=hardblend.safetensors training', '--save_model_as=safetensors', '--network_module=networks.lora', '--text_encoder_lr=5e-05', '--unet_lr=0.0001', '--network_dim=8', '--output_name=JanaDefiNigel_HarrLora', '--lr_scheduler_num_cycles=1', '--learning_rate=0.0001', '--lr_scheduler=cosine', '--lr_warmup_steps=1100', '--train_batch_size=1', '--max_train_steps=11000', '--save_every_n_epochs=1', '--mixed_precision=fp16', '--save_precision=fp16', '--cache_latents', '--optimizer_type=AdamW8bit', '--max_data_loader_n_workers=0', '--bucket_reso_steps=64', '--xformers', '--bucket_no_upscale']' returned non-zero exit status 1.`

[MachineTrooper]

By any chance, did you download/install Kohya initially using a different operating system?

[sgtsixpack]

I installed on windows 10 22h2. I have an AMD 6800xt.

[MachineTrooper]

Well, I guess that has nothing to do with it. I'm really stumped.