genext2fs is painfully slow for multi-GB input

[josch]

Hi,

I'm now using genext2fs with multi-GB tarballs as input. While this works well it also takes several hours on my machine. So I profiled genext2fs:
gprof.txt

If I read the profiling output correctly, then most time is spent in the function allocate().

Do you have any ideas how to improve the speed by introducing better data structures?

[bestouff]

Hi @josch,

indeed I have some ideas to mitigate this; I'm currently a bit short on time but I may try something.
Do you have an easy way of reproducing the problem ?

[josch]

The "easy" way is just to throw a big tarball at it. 😄

For example here is a big system image: https://mister-muffin.de/reform/target-userland-full.tar

[gelrom]

Any luck looking into this?

I've hit this issue as well. For me with a ~10gb tar it seems to basically never complete (on a very powerful machine).
vs e.g. virt-make-fs taking ~30 min.

[gelrom]

Some quick benchmarks that I did make me think there is something highly nonlinear going on:
100mb ~1s
500mb ~10s
800mb ~27s
900mb ~71s
1gb ~130s

note: these were done with a tar of a single file of the above sizes.

[josch]

I observed the same non-linear behavior. Since this is breaking my use-case for genext2fs I instead worked on a patch for e2fsprogs that would allow it to use a tarball as input: tytso/e2fsprogs#118

[pamolloy]

I'm trying to build a 8G image using genimage and genext2fs -d ... has been running for at least 30 minutes. I haven't managed to get it to finish yet. I tried using -a rootfs.tar and ran into a locale issue with a downloaded tar and a segfault on a tar I created.

[pamolloy]

Switched to mke2fs using use-mke2fs = "true" in my genimage.cfg, which seems to perform without issue and complete in less than a minute

[josch]

@pamolloy did the local issue look something like this:

archive_read_next_header(): Pathname can't be converted from UTF-8 to current locale.

If yes, maybe try out #30 and tell me if that fixes your issue?

As for the slowness, I do not know how to fix genext2fs but if you want tarball input, then maybe tytso/e2fsprogs#118 is of interest to you?