[ZEPPELIN-5969] Remove Hadoop2 and move to Hadoop3 shaded client

[pan3793]

What is this PR for?

This PR is based on #4674.

This PR aligns Hadoop dependencies version (3.3.6) in all Zeppelin modules, additionally, it switches to Hadoop shaded client (introduced in HADOOP-11804 since Hadoop 3.0.0) which is recommended by Hadoop, Spark also switched to Hadoop shaded client in SPARK-33212 (3.2.0), which dramatically reduce the Hadoop dependency management efforts, basically, only 3 hadoop jars are required now:

• hadoop-client-api
• hadoop-client-runtime
• hadoop-client-minicluster (only for testing)

There is a JDK 10 compatible issue fixed in HIVE-21508 (Hive 2.3.7), so this PR also upgrades Flink's Hive version from 2.3.4 to 2.3.7. Hive 2.3.7 is chosen because Flink supports Hive 2.3.8 and 2.3.9 in FLINK-26739 (1.16.0) while Zeppelin supports Flink 1.15~1.17.

What type of PR is it?

Improvement

What is the Jira issue?

ZEPPELIN-5969

How should this be tested?

The test cases are modified corresponding, and all GHA tests are passed

Screenshots (if appropriate)

Questions:

• Does the license files need to update? No
• Is there breaking changes for older versions? Yes
• Does this needs documentation? Yes

[matthias-koch]

Suggested change

	<dependency>
	<dependency>
	<groupId>org.apache.hadoop</groupId>
	<artifactId>hadoop-client-api</artifactId>
	<version>${hadoop.version}</version>
	<scope>compile</scope>
	</dependency>
	<dependency>

Without the dependency on hadoop-client-api, r.ir and r.shiny throw an exception upon execution: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream.

[pan3793]

Thanks for checking. I think hadoop-client-runtime will pull hadoop-client-api, thus there should be no problem.

This PR is not ready (I'm a little busy these days, I may can continue this work next week)

[matthias-koch]

hadoop-client-api is a runtime dependency of hadoop-client-runtime, so maven-shaded-plugin should include it. However, I have created the R interpreter package on my local machine and without adding the dependency explicitly, hadoop-client-api is not present in the shaded jar. (But maybe I am doing somthing wrong.)

[Reamer]

How do you know that the maven-shade plugin also adds runtime dependencies? After a first overview I think that the dependency hadoop-client-api has to be added manually.

[matthias-koch]

The description at https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html says:

Requires dependency resolution of artifacts in scope: runtime

But I am really not sure, how all dependencies are resolved.

[pan3793]

alright, I know where the issue comes from.

Do we want to respect the hadoop.deps.scope if we switch to the Hadoop Shaded client?

Note: Spark drops such profiles and always ships Hadoop Shaded client jars into the binary artifacts.

<hadoop.deps.scope>provided</hadoop.deps.scope>
...
<profile>
  <id>include-hadoop</id>
  <properties>
    <hadoop.deps.scope>compile</hadoop.deps.scope>
  </properties>
</profile>

[Reamer]

The main reason for the introduction, I believe, was the interpreter launch mode YARN. See #3786
I don't use this setup myself. As far as I understand it correctly, YARN has already loaded the Hadoop dependencies and therefore they don't need to be in Zeppelin additionally. Perhaps @zjffdu can say more about this.

I am of the opinion that we should include the shaded Hadoop dependency, as this is not available in Kubernetes, Docker, for example. However, we should make sure that we do not deliver the library more than once (e.g. additionally in plugins See #3817)

[pan3793]

As far as I understand it correctly, YARN has already loaded the Hadoop dependencies and therefore they don't need to be in Zeppelin additionally.

There is a switch in YARN to enable/disable Hadoop class population for containers.

we should make sure that we do not deliver the library more than once

QQ, I understand we should not include Hadoop classes in plugins, because they will be loaded into the same JVM with Zeppelin server, so that they can share the Hadoop classes. What about the interpreteres? I assume the interpreters are always run in dedicated JVMs, so Hadoop classes seem always necessary (except for those runtimes who already provided Hadoop classes, e.g. Spark, Flink)?

[Reamer]

There is a switch in YARN to enable/disable Hadoop class population for containers.

I don't know how this is used in Zeppelin.

QQ, I understand we should not include Hadoop classes in plugins, because they will be loaded into the same JVM with Zeppelin server, so that they can share the Hadoop classes. What about the interpreteres? I assume the interpreters are always run in dedicated JVMs, so Hadoop classes seem always necessary (except for those runtimes who already provided Hadoop classes, e.g. Spark, Flink)?

Correct the Zeppelin server & the zengine use the same JVM as the Zeppelin plugins.
In my opinion, the interpreters usually run in separate JVM instances. We should set the scope of Hadoop to Provided in the interpreter, because I think the Hadoop code in the interpreter is only in use for YARN. See

zeppelin/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/YarnUtils.java

Line 20 in 56da029

import org.apache.hadoop.conf.Configuration;

Maybe there will be a way to remove the dependency at some point.

[pan3793]

I revised this question, the rlang interpreter can work both independently and combine with SparkR, for independent case, Hadoop deps are not required, for SparkR case, the Hadoop jars are handled by Spark runtime, either ship Hadoop jars in SPARK_HOME/jars, or setup HADOOP_CLASSPATH properly, so the rlang interpreter don't need to ship Hadoop deps

[pan3793]

I basically have fixed all compile and test issues, the next step is to split it into several small PRs to speed up the review process.

I think we should start with the interpreter modules one by one, and then zengine, server and other modules, eventually dropping the hadoop2 profile and updating docs.

@Reamer could you give some advice?

[Reamer]

I would prefer a larger PR. Where individual tasks are contained in commits. It was clear that the drop of Hadoop2 is very large. Thank you for your work so far.

I think it's great that you have deleted all the excludes in the parent pom.xml, that makes the file much more readable.

Btw. I do not insist on co-authorship.

[pan3793]

unfortunately, I found the IT does not run properly now, see #4699, we may need to postpone this PR after recovering IT

[pan3793]

Hive's transitive deps management is crazy, the exclusion list is borrowed from Apache Flink's pom.xml

[pan3793]

@Reamer it's ready for review, please take a look when you have time

[pan3793]

Hadoop 3.2 is EOL, from the community perspective, we'd better choose a supported version.
https://lists.apache.org/thread/ybygrhvqok0f44s7yzm6c24n0b0s727s

[pan3793]

another reason is, we use the hadoop-client-minicluster to launch Mini YARN cluster for testing in some modules, while since Spark 3.2.0, the pre-built Spark tgz ships Hadoop 3.3.x jars, Hadoop requires that client version <= server version, so we must use Hadoop 3.3 to launch Mini YARN cluster to allow Spark works properly.

[Reamer]

I really like this PR. It removes a lot of unnecessary legacy from the Zeppelin code. I have set a few comments that I would like to have clarified before the merge.

[pan3793]

The Python 3.8 test failure should be addressed in #4748

[pan3793]

@Reamer all failed tests are known flaky tests, this patch should be good to go :)

[Reamer]

I will merge the pull request on Wednesday as long as no further comments are received.

[Armadik]

Could this change break the build? I try to collect and get an error

WARN [2024-05-23 17:04:06,762] ({main} WebAppContext.java[doStart]:533) - Failed startup of context o.e.j.w.WebAppContext@4816278d{/,jar:file:///opt/zeppelin/zeppelin-web-0.12.0-SNAPSHOT.war!/,STOPPED}{/opt/zeppelin/zeppelin-web-0.12.0-SNAPSHOT.war} java.io.FileNotFoundException: JAR entry WEB-INF/lib/hadoop-client-api-3.3.6.jar!/ not found in /opt/zeppelin/zeppelin-web-0.12.0-SNAPSHOT.war

[pan3793]

@Armadik mind providing a reproducible step? e.g. build command, start command, OS platform, JDK version, etc.

[Armadik]

I see an error when running the zeppelin.sh script

Ubuntu 22.04.4 LTS
`apt update

apt install -y curl git maven openjdk-11-jdk npm libfontconfig r-base-dev r-cran-evaluate

wget https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.6.3/apache-maven-3.6.3-bin.tar.gz

sudo tar -zxf apache-maven-3.6.3-bin.tar.gz -C /usr/local/

sudo ln -s /usr/local/apache-maven-3.6.3/bin/mvn /usr/local/bin/mvn

cd Documents/

git clone https://github.com/apache/zeppelin.git

cd zeppelin/

export MAVEN_OPTS="-Xms1024M -Xmx4096M -XX:MaxMetaspaceSize=1024m -XX:-UseGCOverheadLimit -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=war"

./mvnw -B package -DskipTests -Pbuild-distr -Pspark-3.3 -Pinclude-hadoop -Phadoop3 -Pspark-scala-2.12 -Pweb-angular -Pweb-dist -pl '!groovy,!submarine,!flink,!cassandra,!jdbc,!bigquery,!alluxio,!mongodb,!neo4j' -am --no-transfer-progress
`

[Armadik]

] Copying webapp resources [/home/micha/Documents/zeppelin/zeppelin-web/dist]
[INFO] deleting outdated resource WEB-INF/lib/hadoop-client-api-3.3.6.jar
[INFO] deleting outdated resource WEB-INF/lib/hadoop-client-runtime-3.3.6.jar
[INFO] Building war: /zeppelin/zeppelin-web/target/zeppelin-web-0.12.0-SNAPSHOT.war``

This one doesn't seem to work

The team helped me
zip -d /opt/zeppelin/zeppelin-web-0.12.0-SNAPSHOT.war WEB-INF/lib/*

[pan3793]

@Armadik sorry, can not reproduce, both classic and new UI are good on my side.

[Armadik]

I tried a clean build. It seems the problem was in my environment(