Faster dev cycle

[davidmorgan]

I've been digging into build_runner performance, and in particular, scalability of analyzer-based generators as the codebase grows.

It looks like there is room for significant improvement, so that's good :)

It's complicated, so I will need to move carefully with details discussed as I go.

First job will be to add some benchmarks #1335.

[davidmorgan]

Thanks @tenhobi, thanks @rrousselGit.

I'm going to start marking comments I've replied to on this thread as "resolved" to keep it focused, since lots of people are watching it :) but I do appreciate the input.

Caching can certainly help, but it doesn't address the underlying problems, which are 1) performance, 2) granularity of invalidation.

I've created a sub-issue #3810 for granular invalidation, Remi I'll reply to your question there :)

I've also moved performance to a sub-issue #3811 so we can use this issue for overarching things.

[davidmorgan]

For anyone who wants to follow PR by PR, they will be on the sub-issue for scalable performance #3811 for now :)

[davidmorgan]

The fastest proof of concept I have so far is about 6x faster on 1000 files, 120s incremental build -> 20s incremental build.

But I'm still laying the groundwork to make that landable, as I said, it's complicated :)

The benchmark right now at commit 7995cdd178 is

json_serializable
libraries	clean/ms	no changes/ms	incremental/ms
1	23021	3838	5415
100	23047	4067	7286
250	28944	4462	12391
500	40523	5944	29306
750	54127	7291	55905
1000	82186	10385	93929

which at 120s->94s is only 1.25x faster ;)

The code I'm merging is not actually live, needs BuildAssetUriResolver.sharedInstance changing to the new AnalysisDriverModel to use it; I'm considering whether there should be a temporary use-experimental-resolver flag to make it easier to try+compare. But anyway it should be more than 1.25x faster first ;)

[davidmorgan]

@gmpassos thanks! I've created a discussion for people who want to try and give feedback on the work-in-progress performance changes, I'll reply there.