perf: Fix performance issue when assigning previous version

[chme]

I still encounter severe performance problems on a commit graph that has a lot of merge commits (each feature is developed in a feature branch; a version can have more than 20 merge commits - sorry for not testing this as part of #76).

The problematic code path is now in build._assign_previous_versions. It has the same issue that was fixed with #76 for grouping commits into versions.
The commit graph is traversed through all possible branches. Branching/merge commits result in the same commit being checked multiple times.

With large commit graphs, I think, it is best to optimize it, to only follow the commit graph once and gather all information necessary. This PR will find the previous version information during the grouping of the commits.

I will do some more tests tomorrow, but would appreciate, if you could take a look at the proposed solution.

[pawamoy]

I'll trust you again on this one. What would be nice is that we add fuzzing to the test suite: a test that runs the code on a big temp repo, with hundreds of commits, dozens of branches, merge commits, etc., all randomly generated, just to see if the code finishes running in a small amount of time.

[chme]

I added a "test" that generates profiling data with cProfile. It has no assertions, not sure if it should have any ...
The git repo is just a bunch of merges of one commit into main, but it illustrates the current issue well (see below).

I did run the test on different versions of the code:

No. merge commit	2.4.1		PR 78		2.4.0
	f calls	seconds	f calls	seconds	f calls	seconds
15	394,968	0.361	1,764	0.009	1,544	0.007
17	1,574,774	4.413	1,922	0.008	1,674	0.008
18	3,147,717	17.444	2,001	0.008	1,739	0.008

Note: the 2.4.0 version was not the "real" 2.4.0, but I just reverted the changes to build.py. And 2.4.1 was in reality the current main branch.

Running the test with 20 merges was unbearable with current main (I did not wait until it finished and aborted it ...).

My tests on a "real" git repo were successful with this PR applied. git-changelog was fast and it felt as if the response was instant.

For reference, here the generated files:

perf_stats_2.4.0_15.txt
perf_stats_2.4.0_17.txt
perf_stats_2.4.0_18.txt
perf_stats_now_15.txt
perf_stats_now_17.txt
perf_stats_now_18.txt
perf_stats_pr78_15.txt
perf_stats_pr78_17.txt
perf_stats_pr78_18.txt

[pawamoy]

Amazing, thanks a lot!

Since it's not really a test, I'd move this profiling code to a task, in duties.py. Let me do that tomorrow 🙂

[pawamoy]

Actually it's using Pytest fixtures so it could be hard to move that outside of a test run. So instead I think we can simply add a timeout to the test to make sure that it runs fast. WDYT?

[pawamoy]

I rewrote it as a task. Lets merge it as is, and we can refine it later if needed.

[pawamoy]

Thanks a lot for your hard work! 🚀