[ZEPPELIN-6157] Download artifacts from CDN if available #4901
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Error seems to be unrelated, happens in other runs: |
|
I have had the experience that download servers which return the lua script were not available. What is important to me is a stable download and a stable CI environment. Speed is of secondary importance. |
asfgit
pushed a commit
that referenced
this pull request
Dec 19, 2024
## What changes were proposed in this pull request? Current artifacts available in CDN (`dlcdn.apache.org`) may get removed without notice when new releases appear. To avoid broken links, build scripts contain permanent addresses from `archive.apache.org`. But download from `archive.apache.org` may be slow: ``` Thu, 05 Dec 2024 08:39:53 GMT [INFO] --- download:1.6.0:wget (download-sparkr-files) <at> r --- Thu, 05 Dec 2024 08:39:54 GMT Warning: No signatures were supplied, skipping file validation Thu, 05 Dec 2024 08:39:54 GMT [INFO] Read Timeout is set to 60000 milliseconds (apprx 1 minutes) Thu, 05 Dec 2024 08:45:46 GMT [INFO] Expanding: /home/runner/work/zeppelin/zeppelin/rlang/target/spark-3.5.3-bin-without-hadoop.tgz into /home/runner/work/zeppelin/zeppelin/rlang/target ``` Apache Infra's [`closer.lua` script](https://infra.apache.org/release-download-pages.html#closer) can redirect to CDN or archive, depending on artifact availability. This change replaces `archive.apache.org` URLs, and one instance of `dist.apache.org`, with their `closer.lua` equivalent. Output filename has to be specified for `wget` unfortunately. https://issues.apache.org/jira/browse/ZEPPELIN-6157 ## How was this patch tested? Tried some of the URLs locally, both from CLI (`curl -L --head`) and regular build (`mvn -DskipTests clean package`). Full CI: - quick: https://github.com/adoroszlai/zeppelin/actions/runs/12319072153 - frontend: https://github.com/adoroszlai/zeppelin/actions/runs/12319072142 - core: https://github.com/adoroszlai/zeppelin/actions/runs/12319072156 Closes #4901 from adoroszlai/ZEPPELIN-6157. Signed-off-by: Philipp Dallig <[email protected]>
|
Merged into master/branch-0.12 |
|
Thanks @Reamer for reviewing and merging this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Current artifacts available in CDN (
dlcdn.apache.org) may get removed without notice when new releases appear. To avoid broken links, build scripts contain permanent addresses fromarchive.apache.org. But download fromarchive.apache.orgmay be slow:Apache Infra's
closer.luascript can redirect to CDN or archive, depending on artifact availability.This change replaces
archive.apache.orgURLs, and one instance ofdist.apache.org, with theircloser.luaequivalent. Output filename has to be specified forwgetunfortunately.https://issues.apache.org/jira/browse/ZEPPELIN-6157
How was this patch tested?
Tried some of the URLs locally, both from CLI (
curl -L --head) and regular build (mvn -DskipTests clean package).Full CI: