" RuntimeError: User specified autocast device_type must be 'cuda' or 'cpu' " when training. #1567
Replies: 4 comments 3 replies
-
|
Anyone able to help me on this? |
Beta Was this translation helpful? Give feedback.
-
|
Same issue here. M1 pro Max 64GB. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I have:
Launch command:
now i get: ` Traceback (most recent call last): ` This can probably be fixed with enabling the MPS backend of pytorch IMPORTANT NOTE: If --optimizer_type="", the training starts! But this is above my capabilities ;) Cheers |
Beta Was this translation helpful? Give feedback.
-
|
Quick update. This command starts the training: with |
Beta Was this translation helpful? Give feedback.
-
|
somehow the training loss stays around 0.14. doesn't matter if max_train_steps="100" or "1000". |
Beta Was this translation helpful? Give feedback.
-
|
Same here, I'm using Mac M3 . Anyone know how to fix? |
Beta Was this translation helpful? Give feedback.
-
|
your using a device that wants/must run in full precision FP32 autocast. (like my tesla m40) enable "memory efficient attention" option, disable xformers attention option (set "cross attention" to none). make sure save_precision/mixed_precision are FP16 or float. (no bf16 or 8-bit optimizer, except adamw8bit scheduler optimizer) enable the "full fp16 training (experimental)" option. autocast can downcast to 16-bit as long as everything else is set right, which should allow adamw8bit to run on cpu without throwing that error, or causing disabling autocast-cpu from throwing the error from FP32 and adamw8bit trying to run on the same device... mps is trying to emulate nvidia autocast, and this works for older nvidia cards that utilize autocast, so it's worth a try |
Beta Was this translation helpful? Give feedback.
-
When I attempt to train my Lora model (Start training) after setting all parameters, it begins to work for a minute and then gives me this error
RuntimeError: User specified autocast device_type must be 'cuda' or 'cpu'I'm using an M1 Macbook pro, 2021, choosing Stable diffusion v1-5-pruned.safetensors as my model.
I'm using 17 images for the training and all images have been found
This is the traceback:
Traceback (most recent call last): File "/Users/I585070/Documents/Kohya_ss/kohya_ss/venv/bin/accelerate", line 8, in <module> sys.exit(main()) File "/Users/I585070/Documents/Kohya_ss/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/Users/I585070/Documents/Kohya_ss/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 986, in launch_command simple_launcher(args) File "/Users/I585070/Documents/Kohya_ss/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 628, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/Users/I585070/Documents/Kohya_ss/kohya_ss/venv/bin/python', './train_db.py', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--pretrained_model_name_or_path=/Users/I585070/Documents/STABLE_DIFFUSION_WEBUI/stable-diffusion-webui/models/Stable-diffusion/v1-5-pruned.safetensors', '--train_data_dir=/Users/I585070/Downloads/Lora_Training/images', '--resolution=768,768', '--output_dir=/Users/I585070/Downloads/Lora_Training/model', '--logging_dir=/Users/I585070/Downloads/Lora_Training/log', '--save_model_as=safetensors', '--output_name=TShirts', '--lr_scheduler_num_cycles=10', '--max_data_loader_n_workers=0', '--learning_rate=1e-05', '--lr_scheduler=cosine', '--lr_warmup_steps=289', '--train_batch_size=1', '--max_train_steps=2890', '--save_every_n_epochs=1', '--mixed_precision=fp16', '--save_precision=fp16', '--cache_latents', '--optimizer_type=AdamW8bit', '--max_data_loader_n_workers=0', '--bucket_reso_steps=64', '--xformers', '--bucket_no_upscale', '--noise_offset=0.0']' returned non-zero exit status 1.Beta Was this translation helpful? Give feedback.
All reactions