App freezes for large data passed to ui.table.from_pandas

[LimaVsC]

Question

When uploading a large CSV file (4000 rows x 521 columns, in my case), the app freezes, and the on_upload function is not executed.
Here’s an example that reproduces the issue:

from io import StringIO
from nicegui import ui
import pandas as pd
from starlette.formparsers import MultiPartParser

MultiPartParser.max_file_size = 1024 * 1024 * 512


def func(e):
    ui.notify("inside func")
    with StringIO(e.content.read().decode("utf-8")) as f:
        df = pd.read_csv(f, engine="python", sep=None)
    ui.table.from_pandas(df)


ui.upload(on_upload=func)

ui.run()

[rodja]

You'll need to do the reading of the file async: https://github.com/zauberzeug/nicegui/wiki/FAQs#why-is-my-long-running-function-blocking-ui-updates

[LimaVsC]

I don't know how to use async, so with the help of ChatGPT, this code works:

from io import StringIO
from nicegui import ui
import pandas as pd
import asyncio
from starlette.formparsers import MultiPartParser
MultiPartParser.max_file_size = 1024 * 1024 * 512


async def func(e):
    ui.notify("start")

    content = await asyncio.to_thread(e.content.read)

    with StringIO(content.decode("utf-8")) as f:
        ui.notify("reading")
        df = pd.read_csv(f, engine="python", sep=None)
    table_container.clear

    with table_container:
        ui.table.from_pandas(df)
    ui.notify("finish")

upload = ui.upload(on_upload=func)
table_container = ui.column()
ui.run()

However, the page consumes all my RAM. And ui.notify("reading") and ui.notify("finish") never show up.

[falkoschindler]

Using NiceGUI's run.cpu_bound, you can write something like this:

from nicegui import run, ui

def read_csv(content):
    with StringIO(content.read().decode("utf-8")) as f:
        return pd.read_csv(f, engine="python", sep=None)

async def handle_upload(e):
    ui.notify("inside func")
    df = await run.cpu_bound(read_csv, e.content)
    ui.table.from_pandas(df)

ui.upload(on_upload=handle_upload)

ui.run()

[LimaVsC]

Same behavior. Additionally, sometimes the table does not appear. When the table does show up, the interface becomes unresponsive.

[rodja]

Does it work in other python programs (without NiceGUI) like a FastAPI app or a plain Python script?

[LimaVsC]

Yes, it works perfectly in a plain Python script. The code below reproduces a file equivalent to the one I am trying to use:

import numpy as np
import pymc as pm

np.random.seed(42)
true_mu = 5.0
true_sigma = 2.0
n_samples = 500

normal = np.random.normal(loc=true_mu, scale=true_sigma, size=n_samples)
noise = np.random.normal(0, 0.5, size=n_samples)
data = normal + noise
with pm.Model() as model:
    mu = pm.Normal("mu", mu=0, sigma=10)
    sigma = pm.HalfNormal("sigma", sigma=10)
    y_obs = pm.Normal("y_obs", mu=mu, sigma=sigma, observed=data)
    trace = pm.sample(chains=4, idata_kwargs={"log_likelihood": True})

df = trace.to_dataframe()
df.to_csv("trace.csv", index=False)

[rodja]

Thanks for providing the example to generate such large csv files (~38 mb) @LimaVsC. With it I could build a small demo which does not use the upload at all but still freezes:

from nicegui import run, ui
import pandas as pd
from pathlib import Path
from io import StringIO
import time

def read_csv():
    with StringIO(Path("trace.csv").read_bytes().decode("utf-8")) as f:
        return pd.read_csv(f, engine="python", sep=None)

async def load():
    status = ui.notification("reading csv")
    df = await run.cpu_bound(read_csv)
    print(df)
    status.message = "creating ui.table"
    ui.table.from_pandas(df)
    status.dismiss()

ui.timer(1, load, once=True)
alive_indicator = ui.label()
ui.timer(0.1, lambda: alive_indicator.set_text(f"time: {time.monotonic():.1f}s"))

ui.run()

We need to make ui.table.from_pandas async.

[rodja]

The upload works fine as long as the reading is done in a background process:

from nicegui import run, ui
import pandas as pd
from io import StringIO

def create_dataframe(content):
    with StringIO(content.decode("utf-8")) as f:
        return pd.read_csv(f, engine="python", sep=None)

async def load(e):
    status = ui.notification("creating dataframe")
    df = await run.cpu_bound(create_dataframe, e.content.read())
    status.dismiss()
    print(df)

ui.upload(on_upload=load, auto_upload=True)

ui.run()

I'll update the title accordingly.

[rodja]

Many other large dataframes seem to work fine. So for a full reproduction I currently have this (not so minimal) code:

import asyncio
import time
from io import StringIO
import numpy as np
import pandas as pd
import pymc as pm

from nicegui import app, run, ui

def create_df():
    np.random.seed(42)
    true_mu = 5.0
    true_sigma = 2.0
    n_samples = 500

    normal = np.random.normal(loc=true_mu, scale=true_sigma, size=n_samples)
    noise = np.random.normal(0, 0.5, size=n_samples)
    data = normal + noise
    with pm.Model():
        mu = pm.Normal("mu", mu=0, sigma=10)
        sigma = pm.HalfNormal("sigma", sigma=10)
        y_obs = pm.Normal("y_obs", mu=mu, sigma=sigma, observed=data)
        trace = pm.sample(chains=4, idata_kwargs={"log_likelihood": True})

    # NOTE: convert to csv and back to df to make it serializable for ui.table
    df = trace.to_dataframe()
    buffer = StringIO()
    df.to_csv(buffer, index=False)
    buffer.seek(0)
    return pd.read_csv(buffer)

async def load():
    status = ui.notification("creating data")
    df = await run.cpu_bound(create_df)
    print(df)
    status.message = "creating ui.table"
    ui.table.from_pandas(df)
    status.dismiss()

ui.timer(1, load, once=True)
alive_indicator = ui.label()
ui.timer(0.1, lambda: alive_indicator.set_text(f"time: {time.monotonic():.1f}s"))

def startup():
    loop = asyncio.get_running_loop()
    loop.set_debug(True)
    loop.slow_callback_duration = 0.05

app.on_startup(startup)

ui.run()

[falkoschindler]

I experimented with pm.sample(draws=10, ...) (instead of the default 1000) and observed an update message length of around 1.3M. Removing the y_obs variable reduces the message length to 40K. I'm not sure about the impact of y_obs and how it affects the data shown in the table, but it seems to tremendously expand the amount of data sent to the client.

[LimaVsC]

This code fits a Gaussian model to data. Removing y_obs essentially means there is no model (simplifying things here). Additionally, setting draws=10 has no statistical significance. I use this code because it reproduces the DataFrame I’m testing.

I want to be able to use larger files (360+ MB). I’m building a GUI to visualize the data so that I don’t need to create a Python file every time I want to plot a histogram.

[falkoschindler]

Adding y_obs includes a column per observation, drastically increasing the number of cells. For @rodja's example this results in 4000 columns and 521 rows. Here's a reproduction which also crashes the browser due to the sheer amount of UI elements:

COLUMNS = 4000
ROWS = 500
ui.table(columns=[{'name': f'column{c}', 'label': f'Column {c}', 'field': f'column{c}'} for c in range(COLUMNS)],
         rows=[{f'column{c}': f'{r},{c}' for c in range(COLUMNS)} for r in range(ROWS)])

I don't think there's much we can do about it. Tables with many rows might scale better using paging, but I'm not aware of similar methods for tables with so many columns.

[LimaVsC]

All right. Thanks for the time.