Skip to content

Commit

Permalink
Optimize sidebar (#619)
Browse files Browse the repository at this point in the history
* Directory structure adjustment (#570)

* [WIP]update engine usage document (#577)

* update engine usage document

* Optimize the control panel instructions

* pic url

* control panel dir adjust

* update

* desc errora

* add glossary doc (#574)

* add glossary doc

* [feat:3871] Linkis website document optimization (#603)

* version update

* small change

* control panel, udf update

* add udf pic

* add advice configuration

* update en doc

* update udf,spark

* update param conf

* update udf, pic

* add advice configuration

* remove microservice division

* [feat-3857]Optimized the Links SDK instructions (#592)

* Optimized the Links SDK instructions

* Optimized the Links SDK instructions

* fix tr error

* update show

* update show

* update table

* update table

* update trino

* update img(en)

* update img(en)

* update en

* build error

* update en

* update uef en

* Added how to register a new microservice document (#584)

* Added how to register a new microservice document
Co-authored-by: Casion <[email protected]>

* optimize sidebar

* optimize sidebar

* update some docs

* update some docs

* update table docs

* fix url bug

* update table

* update table

* update table

* update table

Co-authored-by: aiceflower <[email protected]>
Co-authored-by: peacewong <[email protected]>
Co-authored-by: 成彬彬 <[email protected]>
  • Loading branch information
4 people authored Dec 6, 2022
1 parent 1416878 commit 3c75079
Show file tree
Hide file tree
Showing 522 changed files with 9,551 additions and 4,462 deletions.
20 changes: 10 additions & 10 deletions blog/2022-03-20-openlookeng.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ tags: [engine]
openLooKeng is an "out of the box" engine that supports in-situ analysis of any data, anywhere, including geographically remote data sources. It provides a global view of all data through a SQL 2003 interface. openLooKeng features high availability, auto-scaling, built-in caching and indexing support, providing the reliability needed for enterprise workloads.

openLooKeng is used to support data exploration, ad hoc query and batch processing with near real-time latency of 100+ milliseconds to minutes without moving data. openLooKeng also supports hierarchical deployment, enabling geographically remote openLooKeng clusters to participate in the same query. With its cross-region query plan optimization capabilities, queries involving remote data can achieve near "local" performance.
Linkis implements the OpenLooKeng engine to enable Linkis to have the ability to virtualize data and support the submission of cross-source heterogeneous queries, cross-domain and cross-DC query tasks. As a computing middleware, Linkis can connect more low-level computing and storage components by using OpenLooKeng's connector based on the connectivity capability of Linkis' EngineConn.
Linkis implements the openLooKeng engine to enable Linkis to have the ability to virtualize data and support the submission of cross-source heterogeneous queries, cross-domain and cross-DC query tasks. As a computing middleware, Linkis can connect more low-level computing and storage components by using openLooKeng's connector based on the connectivity capability of Linkis' EngineConn.

## Development implementation
The implementation of OpenLooKeng ec is extended based on the EngineConn Plugin (ECP) of Linkis. Because the OpengLooKeng service supports multiple users to query through the Client, the implementation mode is the implementation mode of the multi-user concurrent engine.
The implementation of openLooKeng ec is extended based on the EngineConn Plugin (ECP) of Linkis. Because the OpengLooKeng service supports multiple users to query through the Client, the implementation mode is the implementation mode of the multi-user concurrent engine.
That is, tasks submitted by multiple users can run in one EC process at the same time, which can greatly reuse EC resources and reduce resource waste. The specific class diagram is as follows:

【Missing picture】

The specific implementation is that OpenLooKengEngineConnExecutor inherits from ConcurrentComputationExecutor, supports multi-user multi-task concurrency, and supports docking to multiple different OpenLooKeng clusters.
The specific implementation is that openLooKengEngineConnExecutor inherits from ConcurrentComputationExecutor, supports multi-user multi-task concurrency, and supports docking to multiple different openLooKeng clusters.
## Architecture
Architecture diagram:
![image](https://user-images.githubusercontent.com/7869972/166736911-c0f50968-3996-40d0-afdf-52b35d4cd71c.png)
Expand All @@ -25,15 +25,15 @@ Architecture diagram:
The task flow diagram is as follows:
![image](https://user-images.githubusercontent.com/7869972/166737177-57f8f84a-b16d-44bd-b7cf-a61fc2cc160c.png)

The capabilities based on Linkis and OpenLooKeng can provide the following capabilities:
- 1. The connection capability of the computing middleware layer based on Linkis allows upper-layer application tools to quickly connect to OpenLooKeng, submit tasks, and obtain logs, progress, and results.
- 2. Based on the public service capability of Linkis, it can complete custom variable substitution, UDF management, etc. for OpenLooKeng's sql
The capabilities based on Linkis and openLooKeng can provide the following capabilities:
- 1. The connection capability of the computing middleware layer based on Linkis allows upper-layer application tools to quickly connect to openLooKeng, submit tasks, and obtain logs, progress, and results.
- 2. Based on the public service capability of Linkis, it can complete custom variable substitution, UDF management, etc. for openLooKeng's sql
- 3. Based on the context capability of Linkis, the results of OpengLooKeng can be passed to downstream ECs such as Spark and Hive for query
- 4. Linkis-based resource management and multi-tenancy capabilities can isolate tasks from tenants and use OpenLooKeng resources
- 4. Linkis-based resource management and multi-tenancy capabilities can isolate tasks from tenants and use openLooKeng resources
- 5. Based on OpengLooKeng's connector capability, the upper-layer application tool can complete the task of submitting cross-source heterogeneous query, cross-domain and cross-DC query type, and get a second-level return.

## Follow-up plans
In the future, the two communities will continue to cooperate and plan to launch the following functions:
- 1.Linkis supports OpenLooKeng on Yarn mode
- 2. Linkis has completed the resource management and control of OpenLooKeng, tasks can now be queued by Linkis, and submitted only when resources are sufficient
- 3. Based on the mixed computing ability of OpenLooKeng, the ability of Linkis Orchestrator is optimized to complete the mixed computing ability between ECs in the subsequent plan.
- 1.Linkis supports openLooKeng on Yarn mode
- 2. Linkis has completed the resource management and control of openLooKeng, tasks can now be queued by Linkis, and submitted only when resources are sufficient
- 3. Based on the mixed computing ability of openLooKeng, the ability of Linkis Orchestrator is optimized to complete the mixed computing ability between ECs in the subsequent plan.
47 changes: 0 additions & 47 deletions community/microservice-division.md

This file was deleted.

155 changes: 155 additions & 0 deletions discuss/project-hisotry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
## Apache Gobblin
- 孵化时间:2017-02-23 ~ 2021-01-20
- 毕业讨论邮件:https://lists.apache.org/thread/9z79b2648bcxy6pw78bfhvq1jkwy0lox
- 孵化信息: https://incubator.apache.org/projects/gobblin.html
- 反馈的问题:


## Apache Ratis
- 孵化时间:2017-01-03 ~ 2021-02-17
- 毕业讨论邮件: https://lists.apache.org/thread/0x2pbbr0khldfo2t8vp49cnl9oz51wcb
- 孵化信息: https://incubator.apache.org/projects/ratis.html
- 反馈的问题:
- 邮件流量问题 Adjust Github nofications settings to use issues@ mailing list


## Apache Pinot
- 孵化时间:2018-10-17 ~ 2021-07-21
- 毕业讨论邮件: https://lists.apache.org/thread/9y63b4fosbocmsj8qlyzg8j4c6166b54
- 孵化信息: https://incubator.apache.org/projects/pinot.html
- 反馈的问题:
- 文档有一些 ASF 软件的品牌问题 https://github.com/prestodb/presto/issues/16271
One other thing you might want to correct, all of the download links to previous version are broken on your download page. [1] The previous version are automatically archived and can be found here. [2] You may also want to consider not calling the convenience binary an “offical binary release”.


## Apache Hop
- 孵化时间:2020-09-24 ~ 2021-12-16
- 毕业讨论邮件: https://lists.apache.org/thread/jj46blkwrhf0l31srtzo5j5dcv7zsqz3
- 孵化信息: https://incubator.apache.org/projects/hop.html
- 反馈的问题:
- 下载区有几个旧版本;这些应该被删除
- 项目描述太长,需优化相关的措辞

## Apache AGE
- 孵化时间:2020-04-29 ~ 2022-05-18
- 毕业讨论邮件: https://lists.apache.org/thread/jvrtvcw1zb8gf286css7tvol94r3qd68
- 孵化信息: https://incubator.apache.org/projects/age.html
- 反馈的问题:
- 三名 PPMC 成员未订阅私人邮件列表。
- 包括一些 PPMC 成员在内的四名 Committer 没有将他们的 apache id 映射到他们的 GitHub id。他们如何做出贡献?
- 没有人加入PPMC

讨论指出的问题比较多,多次发起了


## Apache YuniKorn
- 孵化时间:2020-01-21 ~ 2022-03-16
- 毕业讨论邮件: https://lists.apache.org/thread/o8y4b3mp21f608745ngqqdhc628qtwxf
- 孵化信息: https://incubator.apache.org/projects/yunikorn.html
- 反馈的问题:


## Apache Doris
- 孵化时间:2018-07-18 ~ 2022-06-15
- 毕业讨论邮件: https://lists.apache.org/thread/nbncjl8fdq0bcjop6l4747c8x9c06q00
- 孵化信息: https://incubator.apache.org/projects/doris.html
- 反馈的问题:
- DorisDB 和 StarRocks的侵权问题
- 百度在中国拥有 Doris 商标,该商标是否已转让给 ASF?
- 与 Apache Impala 的关系

注:涉及比较多的商标和License 问题

- 毕业清单 https://cwiki.apache.org/confluence/display/DORIS/Graduation+Check+List

## Apache MXNet

- 孵化时间:2017-01-23 ~ 2022-06
- 毕业讨论邮件: https://lists.apache.org/thread/3rxzjcmo2y457y6r8ohz1j4qv49joyo6
- 孵化信息: https://incubator.apache.org/projects/mxnet.html
- 反馈的问题:
- 不应该使用 Google Analytics,而是使用 Apache 安装的 Matomo 跟踪器。
- 第三方产品名称问题 3rd party product name listed

- - ICLA签署
https://cwiki.apache.org/confluence/display/MXNET/ICLA+Progress



## Apache ShenYu
- 孵化时间:2021-05-03 ~ 2022-06
- 毕业讨论邮件: https://lists.apache.org/thread/cc2myvso0pbyl39yv26bdmzml8jjvpt2
- 孵化信息: https://incubator.apache.org/projects/shenyu.html
- 反馈的问题:
- 主页描述不够精简 最好用简短而清晰的句子来 说明它在做什么
- 网站上提到了 5 个版本,而不是 4 个,在孵化之前的需要标识出 There's no 2.3.0 version release. And the 2.3.0 document is marked legacy now https://github.com/ apache/incubator-shenyu-website/pull/604
- 文档中名字最好带上 Apache 前缀 https://github.com/apache/incubator-shenyu/pull/3568
- 商标问题 \[Branding Issue] Shenyu is still decleared as a project of dromara community https://lists.apache.org/thread/o9bwzkl3cjbrmzx4n36l63xbv0q112px


## Apache DataSketches
- 孵化时间:2019-03-30 ~ 2020-12-16
- 毕业讨论邮件: https://lists.apache.org/thread/kcpo1ngkmh9q3zgxl5vo64m9m62c45m0
- 孵化信息: https://incubator.apache.org/projects/datasketches.html
- 反馈的问题:
- 如果有重要贡献者没有作为初始贡献者加入 podling,或者如果有其他公司实体可以在代码中主张权利,则 podling 必须从这些个人或公司获得 SGA






1. 之前尝试在Jira上提单申请hub账户 但是反馈坐席满了, 推荐使用Github Docker Registry。  这个是否存在合规性问题(官网文档https://incubator.apache.org/guides/distribution.html#docker 是说:
Artifacts need to be placed in https://hub.docker.com/r/apache/<project>)

2. 另外对于容器镜像构建过程中 的Dockfile文件,使用了基础镜像 如centos:7 ,以及通过如yun install unzip 的安装的基础环境软件包, 或则使用curl/ADD/COPY to a destination inside the Docker image. 这些第三方的工具的使用 是否需要单独在LICENSE中进行 罗列说明 。


1. 我们之前申请hub账号权限, 但是反馈seat 坐席满了, 推荐使用Github Docker Registry。  这个是否符合asf docker 镜像分发要求(官网文档https://incubator.apache.org/guides/distribution.html#docker 提到:
Artifacts need to be placed in https://hub.docker.com/r/apache/<project>)

2. 另外对于构建docker 镜像使用的 Dockfile文件,使用了基础镜像 如centos:7 ,以及通过如yun install unzip 的安装的基础环境软件包, 或则使用 ADD/COPY 添加资源 to a destination inside the Docker image. 这些第三方的工具的使用 是否需要单独在LICENSE文件中进行 罗列说明 。


https://issues.apache.org/jira/browse/INFRA-22785

https://issues.apache.org/jira/browse/INFRA-22785
DockerHub is out of seats. Project is using GitHub Registry instead.

About docker image release and license

We applied for the DockerHub permission before[1], but the feedback DockerHub is out of seats. It is recommended to use the Github Docker Registry.
Does this meet the asf docker image distribution requirements (the official website document[2]mentions:
artifacts need to be placed in https://hub.docker.com/r/apache/<project>)

In addition, for the Dockfile file used to build the docker image[3], use the basic image such as centos,
and the basic environment package installed through such as `yun install unzip`,
or use `ADD/COPY` to add resources to a destination inside the Docker image.
Whether the use of these third-party tools needs to be listed separately in the LICENSE file.
If so, for the toolkit installed by the yum command, it seems that the indirect dependency license cannot be listed well.


1. https://issues.apache.org/jira/browse/INFRA-22785
2. https://incubator.apache.org/guides/distribution.html#docker
3. https://github.com/apache/incubator-linkis/blob/dev-1.3.0/linkis-dist/docker/ldh.Dockerfile


We applied for the Docker Hub permission before[1], but the feedback Docker Hub is out of seats.
It is recommended to use the Github Docker Registry.
Does this meet the ASF docker image distribution requirements (The official website document[2]
mentions:artifacts need to be placed in https://hub.docker.com/r/apache/<project>)

In addition, for the Dockfile file used to build the docker image[3], use the basic image such as CentOS,
and the basic environment package installed through such as `yun install unzip`,
or use `ADD/COPY` to add resources to a destination inside the Docker image.
Whether the use of these third-party tools needs to be listed separately in the LICENSE file.
If so, for the toolkit installed by the yum command, it seems that the indirect dependency license cannot be listed well.


1. https://issues.apache.org/jira/browse/INFRA-22785
2. https://incubator.apache.org/guides/distribution.html#docker
3. https://github.com/apache/incubator-linkis/blob/dev-1.3.0/linkis-dist/docker/ldh.Dockerfile

Best Regards!
Apache Linkis(Incubating)
Chen Xia
4 changes: 4 additions & 0 deletions docs/about/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"label": "About Linkis",
"position": 1.0
}
Loading

0 comments on commit 3c75079

Please sign in to comment.