Feature request(?): Dynamically generated images

[mcclure]

I am working on a ratatui audio app, and I want to visualize waveforms. So in my use case I don't have a specific "image", like the ratatui-image examples expect; rather I'll be generating an image::Image at runtime, based on the display size that turns out to be available then.

I have figured out that I can do this by creating an image::Image of the appropriate size, turning it into a ratatui-image with new_protocol, and resizing my image::Image whenever I detect the frame.area() has changed size. I believe this would work.

However, I am greedy. I want more! My current solution is unsatisfying because:

• The documentation stresses that new_resize_protocol is fundamentally superior to new_protocol, having fewer bugs.
• new_resize_protocol is dynamic, whereas my solution is playing eternal catchup; I update every 60 frames per second, but on a screen with a refresh rate greater than 60, while resizing I'm limited to getting a resized or cropped version of my previous frame before I can detect the resize and prepare a new one.

If I understand the documentation correctly, resize_protocol is doing something really fancy where as the VTXXX terminal reports size changes, ratatui-image is generating adjusted resize dynamically at the exact moment it's needed; moreover it appears using the async trick it can do this without blocking the terminal if the generation turns out to be slow. I want that! My "feature request" is:

There should be a way of creating a new_resize_protocol_callback() (or something) that on resizes, rather than resizing an existing image, calls (async/offthread if possible) a user-supplied callback that generates an image::Image on-demand. The user callback should be able to somewhere specify whether this generated image::Image is expected to be used exactly-as-is or if it should be resized (that is, there should be an equivalent of ratatui_image::Resize.

I recognize this is a large feature ask, and it would not be useful for most library users (as the current well-supported use case— loading fixed-size images which ratatui-image resizes— is surely more common). But it would be so cool! :)

[benjajaja]

The docs are indeed a bit confusing, as they have evolved over time. I will get to it and try to fix them up a bit, and provide better guidance to which widgets / paradigm to chose for a particular app. In the meantime:

new_resize_protocol is not really superior, it's just stateful, it tracks the area it is drawn to and resizes itself accordingly, without the user having to hook up all that logic for every app out there. You are correct that if you are generating images, then it will eternally be catching up, one frame behind.

new_protocol is the right choice for your use case. You want to generate an image, which should perfectly match the area you want to render to beforehand. You get this pixel size from the font-size multiplied by the area col/row count. You then would not want that image to be resized again by new_resize_protocol.

If your program has no user input whatsoever, then this should be quite straightforward to do, it doesn't matter if you block the UI thread.

If you do have user input, then it gets a bit more complicated, you should take a look at the async example or how mdfried works with tokio. It boils down to not generating nor encoding images in the UI thread. Because this is so important, there should not be a new_resize_protocol_callback() which would be blocking and lead users down the wrong path.
You want to continuously generate the images (for your already known pixel size), but in a different thread / task than the UI thread (where you do handle user input events).

Recently I skimmed over this sans-io article, which got me thinking if ratatui-image could provide some kind of async functionality (beyond the async example and widget) from Picker. But I have to read a bit deeper into that.