Jackson 2.8.x dependency problem w/ Elasticsearch 2.3.x by managed dependencies of Spring Boot 1.4.0

[agebhar1]

After upgrade from Spring Boot v1.3.5 to v1.4.0 with a embedded Elasticsearch node the application is broken by NoSuchMethodError:

java.lang.NoSuchMethodError: com.fasterxml.jackson.core.base.GeneratorBase.getOutputContext()Lcom/fasterxml/jackson/core/json/JsonWriteContext;
        at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.writeEndRaw(JsonXContentGenerator.java:327) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.writeRawField(JsonXContentGenerator.java:368) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.common.xcontent.XContentBuilder.rawField(XContentBuilder.java:914) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.common.xcontent.XContentHelper.writeRawField(XContentHelper.java:378) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.search.internal.InternalSearchHit.toXContent(InternalSearchHit.java:476) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.search.internal.InternalSearchHits.toXContent(InternalSearchHits.java:184) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.search.internal.InternalSearchResponse.toXContent(InternalSearchResponse.java:111) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.action.search.SearchResponse.toXContent(SearchResponse.java:195) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.rest.action.support.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:43) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.rest.action.support.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:38) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.rest.action.support.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:30) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) [elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) [elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction$2.doRun(SearchQueryThenFetchAsyncAction.java:138) [elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-2.3.4.jar:2.3.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

The problematic code from Elasticsearch (JsonXContentGenerator:327)

public void writeEndRaw() {
    assert base != null : "JsonGenerator should be of instance GeneratorBase but was: " + generator.getClass();
    if (base != null) {
        base.getOutputContext().writeValue();
    }
}

Elasticsearch (v2.3.4) depends on Jackson v2.6.6 and is managed by Spring Boots provided version v2.8.1. In Jackson's GeneratorBase v2.6 the return type is JsonWriteContext

JsonWriteContext | getOutputContext()
Note: co-variant return type.

which has a method writeValue() but in GeneratorBase v2.8 the return type is changed to it's super type JsonStreamContext

JsonStreamContext | getOutputContext()
Note: type was co-variant until Jackson 2.7; reverted back to base type in 2.8 to allow for overriding by subtypes that use custom context type.

The current provided version of Jackson breaks usage of Elasticsearch with out-of-the-box Spring Boots 1.4 managed dependencies.

[agebhar1]

added a simple test case agebhar1/spring-boot-6508-jackson-elasticsearch

[anand1st]

This affects jest-2.0.3 functionality integration with spring-boot-starter-data-elasticsearch as well

[anand1st]

This problem is seen only in the embedded elasticsearch scenario. When used against a standalone elasticsearch, this is not a problem. In my case, I use the embedded elasticsearch for unit testing. One temporary workaround for me was to hardcode the jackson dependencies to version 2.6.6 and change the scope to test so that these older jackson libraries are not transitively passed down.

[jloisel]

This issue also happens with standalone elasticsearch when using elasticsearch java library as client. Downgrading Jackson to version 2.7.5 fixes the issue. This is not a Spring boot issue, it's elasticsearch which should upgrade to Jackson 2.8.x.

[agebhar1]

@jloisel 😕 the latest version of Elasticsearch 2.3.5 uses Jackson 2.6.6 ... as mentioned in #6536 the upcoming Elasticsearch version 5 seems to be using Jackson 2.8.1

[rburawes]

I have the same issue, I'm using spring-data-elasticsearch-2.0.2.RELEASE. I've downgraded my Jackson dependency to 2.7.6 and it solved the issue, at least for now.

[philwebb]

We're stuck between a rock and a hard place with this one. I don't think we want to downgrade our Jackson version just for Elasticsearch but we should document that a downgrade is necessary.

[philwebb]

Perhaps we could also throw a nicer error if we detect an invalid combination.

[agebhar1]

@philwebb Did you have something in mind to accomplish/detect it?

[philwebb]

@agebhar1 I was thinking something in the Elasticsearch auto-configuration could check that the correct Jackson version is being used and fail hard if it's not.

[wilkinsona]

We're going to drop back to Jackson 2.7.x and add a test that checks that Jackson and Elasticsearch are playing nicely together.

[wilkinsona]

@jhoeller Is there any problem with dropping back to 2.7.x from the framework's perspective?

[jhoeller]

Nope, we support Jackson 2.6+ and just recommend 2.7.x or - for the ambitious - 2.8.x at this point.

[agebhar1]

We're going to drop back to Jackson 2.7.x and add a test that checks that Jackson and Elasticsearch are playing nicely together.

@wilkinsona I already create a simple test case (see 2nd comment) which I can contribute. Should the test case one Java file or should it be included somewhere else?

[cowtowncoder]

Apologies for breakage here. I was hoping that co-variant return type would work in this case, but realize now that it does not (since method signature is still different even if there are no intermediate assignments). If anyone can suggest something to help with 2.8.2 I would be happy to add a patch, but am not sure there is much that can be done.

[philwebb]

@cowtowncoder I can't see any obvious way to fix it. Hopefully Elasticsearch can provide a patch.

[cowtowncoder]

@philwebb Unfortunately it is a tricky problem, and I underestimated part of binary incompatibility issue and only solved source-compatibility part in the end.
The reason for change itself has to do with non-JSON data formats (possibly the new Properties backend) needing a custom subtype. But I wish I had figured out a more robust way to change this.

[wilkinsona]

We need to remove the dependency management for jackson-datatype-jaxrs which is new in 2.8

[GaganNoorSingh]

Added jackson Dependency to build.sbt works fine now
"com.fasterxml.jackson.dataformat" % "jackson-dataformat-smile" % "2.8.2"

[alexzhang2015]

cool ...