Bug Report: vtgate planning performance regression for big "union" from v15 to v19

[czhang1216]

Overview of the Issue

Continuation from the slack thread. After switching from v15 to v19, we are seeing 2x-3x+ 2-30x planning performance regression on some particular query. This regression is significant when small amount of data is read.

Reproduction Steps

I included our schema and query in question in attached patches. Apply them to v15 and v19 repo, you'll be able to see the performance difference.

Step-by-step reproduce instructions:

• Please download 15.patch and 19.patch to an empty directory, then run the following in that directory:

$ git clone https://github.com/vitessio/vitess.git -b release-15.0 --depth=1 vitess15
$ cd vitess15
$ git apply ../15.patch
$ go test -benchmem -run=^$ -bench ^BenchmarkMine$ --benchtime=5s -count=12 vitess.io/vitess/go/vt/vtgate/planbuilder

$ cd ..
$ git clone https://github.com/vitessio/vitess.git -b release-19.0 --depth=1 vitess19
$ cd vitess19
$ git apply ../19.patch
$ go test -benchmem -run=^$ -bench ^BenchmarkMine$ --benchtime=5s -count=12 vitess.io/vitess/go/vt/vtgate/planbuilder
# Looking at "ns/op" 19 takes ~2x time compared to 15.

The query in question roughly looks like the following. As you can see it has a large UNION subquery.

select
    blahblah
from my_table
where
    id in (
        (
            select id
            from my_table
            where blahblah
        )
        UNION ALL
        (
            select id
            from my_table
            where blahblah
        )
        UNION ALL
        # ... in total 50 sub queries

(Please ignore the inefficiency of the query itself, it's just for the purpose of displaying planner benchmark regression.

Binary Version

vtgate version Version: 19.0.8 (Git revision 2665aa2a74fdd26c4606598ba68746be30bd9148 branch 'HEAD') built on Tue Dec  3 05:09:55 UTC 2024 by runner@fv-az1670-25 using go1.22.9 linux/amd64

Operating System and Environment details

Red Hat Enterprise Linux Server 7.9 

$ uname -sr
Linux 3.10.0-1160.80.1.el7.x86_64

Log Fragments

No response

[czhang1216]

Sorry I forgot to attach the patches. They are here:

15.patch
19.patch