flamegraph x-axis

[petergaultney]

My apologies for what is probably an obvious thing.

I suppose technically the x-axis is memory usage, but can you also confirm that it represents, from left-to-right, the order in which functions in the process are called?

Incidentally, if that's the case, it would be super useful for me to see a beginning and end time for each horizontal function call. Perhaps on hover in the HTML page? Obviously they are non-linear with respect to memory usage, but having those timestamps would help a lot.

[itamarst]

The left to right order is not the order in which they're called. It's somewhat arbitrary.

For one thing, multiple calls to the same stack are merged into a single stack in the flamegraph. So you might have callstacks C1, C2, C1, C2 allocating in your code. The resulting flamegraph will have C1, C2, or maybe C2, C1, but in either case you're not getting a view of execution order. And you can imagine for some applications it's "here's (literally) a million little tiny interleaved allocations", not just four large ones.

So given the nature of the visualization, you just can't get meaningful ordering out of a flamegraph.
Might be possible with different, additional visualizations, or reports, though.

Perhaps... a table of N largest allocations, with timestamps? Won't help with lots-of-small-allocations cases, but would help anything that does large allocations.

Separately, I'm working on a commercial performance+memory profiler for production data science/scientific computing batch jobs, and that has the benefit of (potentially) allowing even more informative visualizations because:

1. It'll already be tracking program execution order in general.
2. It's a more constrained problem space, it's focusing even more on batch jobs and large allocations, so e.g. it's using sampling, unlike Fil which tracks all allocations.

[petergaultney]

incidentally, I might be interested in trying out your commercial profiler. I am in fact working on large-ish data science batch jobs, and I think I'd be fine getting sampled, with the one caveat that I'm not 100% sure how pandas and our parquet library handle memory allocation - if they occasionally allocate several GB in one go, and your sampling algorithm doesn't happen to choose that instance, then it might give some pretty misleading results?

I will note that while the current issue I'm debugging doesn't require support for multiprocessing, I do use multiprocessing fairly heavily and in some cases, it might be fairly critical to be able to profile multiple processes at once. Though, if you're taking more of a monitoring approach, maybe you're already on track to solve this somehow?

[itamarst]

1. Sampling is tied to allocation size, so all large allocations will be tracked. Any inaccuracy will be for small allocations... but those are inherently less relevant, so it mostly works out just fine.
2. Multiprocessing is on the todo list, and I have a design, but unfortunately I haven't implemented it yet. But I will get to it.

I will keep you updated on progress (I saw you subscribed for updates?).