Allow custom models

[lrq3000]

Please allow to select custom models, especially the main diffusion rendering one, so we can choose a lightning or turbo model that can render faster in much fewer steps and finetuned for our purpose (eg, realist, anime, etc).

[ai-anchorite]

You can quite easily do this by editing app.py. The model loaders are quite obvious. Drag the file into a coding LLM and will be able to switch them out for another huggingface derived model.

It's the beautify of permissive licences and locally run apps. Once it's on your system, it's yours to do with as you will :)

Not sure I'll be adding a UI model loader here. It's a port of an existing project and the nature of these things is something better will come along and replace it before long!

[lrq3000]

Oh ok I see, thank you for the pointers. Exploring coding LLM is on my todo list, I'll give it a try then.

Would you accept a PR if I manage to implement it? While I agree that there likely will be something better at some point in the future, I don't think it will be replaced anytime soon, it hasn't for now at least for creative upscaling, and ClarityAI exists since at least a year I think?

[ai-anchorite]

Bear in mind that this was a demo project by Finegrain to show that their Refiners library can recreate an existing diffusion task. So it may well not respond as expected to model changes etc.

There's resizing happening in the pre-upscale stage, then does an upscale using ESRGAN prior to the diffusion upscale pass. This affects the expected image output dimensions.

Or something like that. They go into the project detail here: https://blog.finegrain.ai/posts/reproducing-clarity-upscaler

My driving interest in the project was to bring the HF demo to run locally via Pinokio, especially since it worked with 8GB VRAM! And because i enjoy tweaking app UIs more than using apps haha. The actual upscaling logic was left alone. The code in enhancer.py and esrgan_model.py is beyond me.

Switching to an inferior low-step model doesn't seem productive, i think most people would accept a couple of extra seconds than lose output quality? An anime specific model could be interesting though.

There's definitely potential for a much better creative upscaler in here. Personally I dislike SD1.5's unavoidable affect on human/realistic faces, so don't use it myself, but feel free to fork and tweak and improve!