fix(doc): Do not allow duplicate names to be created #511
Conversation
|
Just some commentary, this issue was super hard to track down! I was looking at an issue where the much more populated and well known The documents were identical, except the score showed that matches on the phrase field were being adjusted based on a field length of 4 for the I had to dig into the documents generated by both importers to learn that the difference was in duplicate values in the Looking back, we've often been confused as to why Geonames records for a given admin area seem to be preferred, and this might be the reason! So hopefully results will be much better with this PR and/or pelias/model#132 |
orangejulius
force-pushed
the
do-not-create-duplicate-names
branch
from
231c736 to
876a4f7
Compare
This should help fix some scoring issues identified in #511 While not a complete fix, it should mitigate the effects of pelias/openstreetmap#507 somewhat.
While diagnosing an issue related to scoring, I discovered that WOF records are sometimes created with duplicate name values. While the pelias/model code can detect some of them (and more will be fixed with pelias/model#132), we should fix this issue at the source. Here's an example of what a document might look like today, before this PR: ``` { name: { default: [ 'Kansas City' ] }, phrase: { default: [ 'Kansas City', 'Kansas City' ] }, ... } ``` This can be fixed by checking each potential alternate name against the "primary" name value.
orangejulius
force-pushed
the
do-not-create-duplicate-names
branch
from
876a4f7 to
a6c80a2
Compare
While diagnosing an issue related to scoring, I discovered that WOF records are sometimes created with duplicate name values. While the pelias/model code can detect some of them (and more will be fixed with pelias/model#132), we could also fix this issue at the source.
Here's an example of what a document might look like today, before this PR:
This can be fixed by checking each potential alternate name against the "primary" name value.
We might not want to merge this PR, since it only fixes the issue in this repo, but it might also be nice to test this change in a single repository first.