SOLR-17274: allow JSON atomic updates to use multiple modifiers or a modifier like 'set' as a field name if child docs not enabled #2451

eukaryote · 2024-05-09T00:14:28Z

Fixes problems with the JsonLoader's parsing of atomic update values that use multiple modifiers in a single update or reference a field name that matches a modifier (like set or add).

https://issues.apache.org/jira/browse/SOLR-17274

Description

Fixes two problems with JSON atomic updates where an atomic update value like {"set": {"set": "foo"}} (set a field named set to value foo) or {"foo": {"add": "bar", "remove": "baz"}} (add bar value to and remove baz values from field foo) would throw an exception due to isChildDoc thinking a nested doc was being updated. The exact exception message was:

Unable to index docs with children: the schema must include definitions for both a uniqueKey field and the '_root_' field, using the exact same fieldType

Solution

I modified the isChildDoc method of JsonLoader to check if the schema is usable for child docs, and to return false if not. I also had to revert a part of the logic in parseObjectFieldValue so that it would return all the modifier values when there are multiple. The isChildDoc behavior is unchanged if the schema does support child docs, so the same error with continue to be thrown in that case.

Tests

There weren't any atomic update tests that used the JSON loader previously, so I added a new schema file with the minimal fields needed to test this functionality and some tests for existing functionality via Json and for what is changed by this PR.

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide

The only tests that failed were the S3 tests, which I believe is common and not necessarily a problem if my changes shouldn't have affected the s3 functionality.

- this modifies `isChildDoc` to always return false if child docs are not possible according to the schema, and restores part of the logic of `parseObjectFieldValue` so that it can handle multiple modifiers - re-enables the ability to use multiple modifiers in a single update, like `{"add": "foo", "remove": "bar"}` - re-enables the ability to have a field with a name like `set` or `remove` and be able to use updates like `{"set": {"set": "foo"}` - both of these are possible now but only if the schema doesn't support child/nested docs; if the schema does support them, then functionality is unchanged and either will continue throwing an exception

dsmiley

Nice test

solr/core/src/test/org/apache/solr/update/processor/AtomicUpdatesJsonTest.java

dsmiley · 2024-05-09T01:04:45Z

solr/core/src/test/org/apache/solr/update/processor/AtomicUpdatesJsonTest.java

+    doc.setField("id", "1");
+    doc.setField(
+        "set",
+        new HashMap<String, String>() {


Map.of doesn't seem to allow null values, so I can't do Map.of("set", null) here. What do you suggest instead of HashMap?

ah well; its okay. I hate the syntax hack with the HashMap subclass but I see its appeal in being able to use it inline like this.

Oh, it looks like one of the checks is failing because of the instance initializer block, so I'll change it to use HashMap but avoid that.

solr/core/src/test/org/apache/solr/update/processor/AtomicUpdatesJsonTest.java

- Map.of doesn't allow null values, so there's still one remaining use of HashMap

dsmiley · 2024-05-10T13:16:06Z

Thinking of wording for CHANGES.txt:

Improved disambiguation of atomic updates from child docs for JSON based updates (not an issue for other formats).
Multiple atomic updates to a field in one update is supported (again). Schema fields can be named "set", "add", etc. (again). These improvements only apply when the schema doesn't have the root field (no child docs).

dsmiley · 2024-05-10T13:17:43Z

Alternatively I rather like the title here:

allow JSON atomic updates to use multiple modifiers or a modifier like 'set' as a field name if child docs not enabled

solr/core/src/test/org/apache/solr/update/processor/AtomicUpdateJsonTest.java

…teJsonTest.java

dsmiley · 2024-05-10T21:56:15Z

I'll merge this tonight unless I hear to the contrary.

…modifier like 'set' as a field name if child docs not enabled (#2451) * fix two problems with json atomic update [SOLR-17274] - this modifies `isChildDoc` to always return false if child docs are not possible according to the schema, and restores part of the logic of `parseObjectFieldValue` so that it can handle multiple modifiers - re-enables the ability to use multiple modifiers in a single update, like `{"add": "foo", "remove": "bar"}` - re-enables the ability to have a field with a name like `set` or `remove` and be able to use updates like `{"set": {"set": "foo"}` - both of these are possible now but only if the schema doesn't support child/nested docs; if the schema does support them, then functionality is unchanged and either will continue throwing an exception --------- Co-authored-by: Calvin Smith <[email protected]> Co-authored-by: David Smiley <[email protected]> (cherry picked from commit 3b30f25)

pasupathi-rajamanickam-odp · 2024-06-26T15:28:01Z

When this will be published for public use?

dsmiley · 2024-06-26T18:29:14Z

Whenever 9.7 is released. Should be within a month.

github-actions bot added tests cat:index labels May 9, 2024

dsmiley reviewed May 9, 2024

View reviewed changes

Calvin Smith added 4 commits May 9, 2024 18:02

use assertFalse and use singular for test file name

38fec30

remove 'indent=true', and all but one uses of HashMap

4887252

- Map.of doesn't allow null values, so there's still one remaining use of HashMap

not needed since there are no nested docs in these tests

25053aa

use delQ instead of copied raw query

b083a77

Calvin Smith added 2 commits May 10, 2024 09:42

no instance initializer (to avoid precommit error)

111641e

comment-only update

b471e18

dsmiley reviewed May 10, 2024

View reviewed changes

solr/core/src/test/org/apache/solr/update/processor/AtomicUpdateJsonTest.java Outdated Show resolved Hide resolved

dsmiley and others added 2 commits May 10, 2024 17:38

Update solr/core/src/test/org/apache/solr/update/processor/AtomicUpda…

7897402

…teJsonTest.java

CHANGES.txt

1f4b4a0

dsmiley merged commit 3b30f25 into apache:main May 11, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SOLR-17274: allow JSON atomic updates to use multiple modifiers or a modifier like 'set' as a field name if child docs not enabled #2451

SOLR-17274: allow JSON atomic updates to use multiple modifiers or a modifier like 'set' as a field name if child docs not enabled #2451

eukaryote commented May 9, 2024 •

edited

Loading

dsmiley left a comment

dsmiley May 9, 2024

eukaryote May 10, 2024

dsmiley May 10, 2024

eukaryote May 10, 2024

eukaryote May 10, 2024

dsmiley commented May 10, 2024

dsmiley commented May 10, 2024

dsmiley commented May 10, 2024

pasupathi-rajamanickam-odp commented Jun 26, 2024

dsmiley commented Jun 26, 2024

SOLR-17274: allow JSON atomic updates to use multiple modifiers or a modifier like 'set' as a field name if child docs not enabled #2451

SOLR-17274: allow JSON atomic updates to use multiple modifiers or a modifier like 'set' as a field name if child docs not enabled #2451

Conversation

eukaryote commented May 9, 2024 • edited Loading

Description

Solution

Tests

Checklist

dsmiley left a comment

Choose a reason for hiding this comment

dsmiley May 9, 2024

Choose a reason for hiding this comment

eukaryote May 10, 2024

Choose a reason for hiding this comment

dsmiley May 10, 2024

Choose a reason for hiding this comment

eukaryote May 10, 2024

Choose a reason for hiding this comment

eukaryote May 10, 2024

Choose a reason for hiding this comment

dsmiley commented May 10, 2024

dsmiley commented May 10, 2024

dsmiley commented May 10, 2024

pasupathi-rajamanickam-odp commented Jun 26, 2024

dsmiley commented Jun 26, 2024

eukaryote commented May 9, 2024 •

edited

Loading