DISCUSS: Default domain compaction threshold is too low for basejdbc derived connectors

[lhofhansl]

I just ran into a scenario where dynamic filtering seemingly did not work.
In this specific case I found it only with a debugger, and it turns out that the domain had translated a small list of values into a range query, and that the few values varied greatly, so that the range ended up being almost the entire table.

Increasing the domain compaction threshold for this connector solved the problem.

The default is 32, and that seems very small!
As a comparison, Phoenix has a similar setting, but it defaults to 50.000.

As stated in the title, it's a discussion. Perhaps it should be more prominently documented so that folks do not have to resort to the debugger :)

[raunaqmorarka]

It is documented in every jdbc based connector, e.g. https://trino.io/docs/current/connector/clickhouse.html#general-configuration-properties
The default is small to be on the safe side. Ideally the default for each connector should be set based on the source system's capabilities, but since we can't reasonably benchmark everything we used a small default.
If you're confident about phoenix being able to handle bigger predicates without any perf regression, feel free to open a PR to increase phoenix connector default for domain compaction.

[lhofhansl]

Thanks @raunaqmorarka .

In this case it is clear what to look for after I found out the name of the config in the source via the debugger. :)
I did not know what to look for; and without the debugger I would have concluded that dynamic filters are ineffective.

I tested with sets up 100k items and in all cases in my particular usecase that was faster than compressing to a range query. 32 seems awfully small. YMMV of course.

Perhaps we can add something here: https://trino.io/docs/current/admin/dynamic-filtering.html?highlight=dynamic%20filters ...
Or perhaps what we have is a good as it gets in terms of documentation.

Re: Phoenix. That was just an example and has nothing to do with the Phoenix connector. Phoenix has a "skip scan limit" after which it will change the plannode to a do rangescan instead. That limit is set to 50k... Just an example. The case I tested was a connector to an internal proprietary database.

In any case, I just wanted to spare others the work I had to do to figure this out. We can close this this discussion.