Fix usage of whitespace tokenizer

[Jade-GG]

Makes it a little more reliable, as the whitespace tokenizer made synonyms case sensitive.

Could also use the standard tokenizer, shouldn't really make a difference here.

Tokenizer reference

[indykoning]

Nice! Just to be sure, lowercase is an extension of whitespace right? So no breaking changes are made

[Jade-GG]

Nice! Just to be sure, lowercase is an extension of whitespace right? So no breaking changes are made

Not exactly, my original usage of the whitespace tokenizer was actually a breaking change:

• whitespace only splits queries when it encounters a whitespace character, e.g. aBc.dEf 4 gHi only gets turned into ['aBc.dEf', '4', 'gHi']
• lowercase will split on any non-letter character, turning the same string into ['abc', 'def', 'ghi']

I do now realize that this means every number put into the search query gets removed... It might be better to just stick to the standard tokenizer even though it doesn't make the query case insensitive, but it would at least fix the accidental breaking change I made.

[indykoning]

That's a shame, might it be a good idea to make this configurable? Or does standard already cover most, if not all use cases we've come across?

[Jade-GG]

That's a shame, might it be a good idea to make this configurable? Or does standard already cover most, if not all use cases we've come across?

It's already kind of configurable if you use the eventy filters and set the mappings/settings manually.

We could also make it configurable globally but then we'd have to figure out a way to apply it to every mapping, rather than just the ones that get the synonym filter applied (which is what this is actually from). Could be an interesting story, but probably for another time 😅

[indykoning]

If they're accessible in the eventy filters that will be enough if someone would want to overwrite this setting for now.
If it happens often in the future we can always make changes to make that easier