Skip to content

Commit

Permalink
elasticsearch doc support (#467)
Browse files Browse the repository at this point in the history
* modify link jump 404 issue
* modify elasticsearch doc
  • Loading branch information
aiceflower authored Aug 8, 2022
1 parent 218811e commit 0a606d7
Show file tree
Hide file tree
Showing 5 changed files with 271 additions and 32 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@ dist-ssr
*.idea
*.iml.
.vscode
package-lock.json
32 changes: 22 additions & 10 deletions docs/deployment/web-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,33 @@
title: Linkis Console Deployment
sidebar_position: 6
---

Linkis 1.0 provides a Linkis Console, which provides functions such as displaying Linkis' global history, modifying user parameters, managing ECM and microservices, etc. Before deploying the front-end management console, you need to deploy the Linkis back-end. Linkis deployment manual See: [Linkis Deployment Manual](deployment/quick-deploy.md)
```
The linkis web service uses nginx as a static resource server. The access request process is as follows :
Linkis Management Console request -> Nginx ip:port-> Linkis-gateway ip:port-> Other services
```
Linkis 1.0 provides a Linkis Console, which provides functions such as displaying Linkis' global history, modifying user parameters, managing ECM and microservices, etc. Before deploying the front-end management console, you need to deploy the Linkis back-end. Linkis deployment manual See: [Linkis Deployment Manual](quick-deploy.md)

## 1. Preparation

1. Download the web installation package from the release page of Linkis ([click here to enter the download page](https://linkis.apache.org/download/main/)), apache-linkis-xxx-incubating-web-bin. tar.gz
Manually decompress: tar -xvf apache-linkis-x.x.x-incubating-web-bin.tar.gz, the decompressed directory is:
Manually decompress: tar -xvf apache-linkis-x.x.x-incubating-web-bin.tar.gz

The decompression directory is as follows.
```
config.sh
dist
install.sh
....
├── config.sh
├── DISCLAIMER
├── dist
├── install.sh
├── LICENSE
├── licenses
└── NOTICE
```

## 2. Deployment
    There are two deployment methods, automated deployment and manual deployment

### 2.1 Automated deployment
    Enter the unzipped front-end directory, and edit ```vi config.sh ``` in the directory
Change the front-end port and back-end interface address, the back-end interface address is the gateway address of Linkis.
#### 2.1.1 modify config.sh file (use vim or nano)

```$xslt
#Configuring front-end ports
Expand All @@ -34,7 +41,12 @@ linkis_url="http://localhost:9001"
linkis_ipaddr=$(ip addr | awk'/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1" , "g", $2)}')
```

After the modification is executed in this directory, you need to use sudo to execute: ```sudo sh install.sh ```
#### 2.1.2 execute deployment script

```shell
#sudo permission is required to install nginx
sudo sh install.sh
```

After execution, you can directly access it in Google browser: ```http://linkis_ipaddr:linkis_port``` where linkis_port is the port configured in config.sh, and linkis_ipaddr is the IP of the installation machine

Expand Down
144 changes: 144 additions & 0 deletions docs/engine_usage/elasticsearch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
---
title: ElasticSearch Engine Usage
sidebar_position: 11
---


This article mainly introduces the configuration, deployment and use of ElasticSearch EngineConn in Linkis1.0.

## 1. Environment configuration before using the ElasticSearch EngineConn

If you want to use the ElasticSearch EngineConn on your server, you need to prepare the ElasticSearch connection information, such as the connection address, user name and password of the ElasticSearch service, etc.

## 2. ElasticSearch engineConn configuration and deployment

### 2.1 ElasticSearch version selection and compilation

Note: The Linkis project needs to be fully compiled before ElasticSearch EngineConn is built.

The published installation and deployment packages do not contain the engine plugin by default.

You can click [Linkis engine installation guide] (https://linkis.apache.org/zh-CN/blog/2022/04/15/how-to-download-engineconn-plugin) to deployment installation, or according to the following process, Manual compilation and deployment

Compile ElasticSearch engine separately

```
${linkis_code_dir}/linkis-enginepconn-lugins/engineconn-plugins/elasticsearch/
mvn clean install
```

### 2.2 ElasticSearch EngineConn deployment and loading

Upload the engine package compiled in Step 2.1
```bash
${linkis_code_dir}/linkis-engineconn-plugins/engineconn-plugins/jdbc/target/out/elasticsearch
```
to the engine directory on the server
```bash
${LINKIS_HOME}/lib/linkis-engineplugins
```

### 2.3 ElasticSearch EngineConn Labels
Linkis1.x works with tags, so you need to insert data into our database as shown below.

The configuration of the management console is managed by engine labels. If a new engine needs to be configured, you need to modify the metadata of the corresponding table.

```
linkis_ps_configuration_config_key: Insert the key and default values for the engine's configuration parameters
linkis_cg_manager_label:Insert engine label such as:elasticsearch-7.6.2
linkis_ps_configuration_category: Insert the directory association of the engine
linkis_ps_configuration_config_value: Insert the configuration that the engine needs to show
linkis_ps_configuration_key_engine_relation:Association between configuration items and engines
```

```sql

SET @ELASTICSEARCHENG_LABEL="elasticsearch-7.6.2";
SET @ELASTICSEARCHENG_ALL=CONCAT('*-*,',@ELASTICSEARCHENG_LABEL);
SET @ELASTICSEARCHENG_IDE=CONCAT('*-IDE,',@ELASTICSEARCHENG_LABEL);

insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType',@ELASTICSEARCHENG_ALL, 'OPTIONAL', 2, now(), now());
insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType',@ELASTICSEARCHENG_IDE, 'OPTIONAL', 2, now(), now());

select @label_id := id from linkis_cg_manager_label where `label_value` = @ELASTICSEARCHENG_IDE;
insert into linkis_ps_configuration_category (`label_id`, `level`) VALUES (@label_id, 2);


INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.elasticsearcheng.url', 'such as:http://127.0.0.1:8080', 'conn address', 'http://127.0.0.1:8080', 'Regex', '^\\s*http://([^:]+)(:\\d+)(/[^\\?]+)?(\\?\\S*)?$', 'elasticsearcheng', 0, 0, 1, 'data source conf');
INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.elasticsearcheng.catalog', 'catalog', 'catalog', 'system', 'None', '', 'elasticsearcheng', 0, 0, 1, 'data source conf');
INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`) VALUES ('linkis.elasticsearcheng.source', 'source', 'source', 'global', 'None', '', 'elasticsearcheng', 0, 0, 1, 'data source conf');


-- elasticsearcheng-*
insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`)
(select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config
INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'elasticsearcheng' and label_value = @ELASTICSEARCHENG_ALL);

-- elasticsearcheng default configuration
insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`)
(select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation
INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @ELASTICSEARCHENG_ALL);

```

### 2.4 ElasticSearch EngineConn configurations

| configurations | default | is necessary |description |
| ------------------------ | ------------------- | ---|---------------------------------------- |
| linkis.es.cluster | 127.0.0.1:9200 | yes |ElasticSearch cluster,separate multiple nodes using commas |
| linkis.es.username | None |no | ElasticSearch cluster username |
| linkis.es.password | None |no | ElasticSearch cluster password |
| linkis.es.auth.cache | false |no | Whether the client is cache authenticated |
| linkis.es.sniffer.enable | false |no | Whether the sniffer is enabled on the client |
| linkis.es.http.method | GET | no | Request methods |
| linkis.es.http.endpoint | /_search |no | the Endpoint in JSON Script |
| linkis.es.sql.endpoint | /_sql |no | the Endpoint in SQL |
| linkis.es.sql.format | {"query":"%s"} | no | the template of SQL script call , %s replaced with SQL as the body of the request request ElasticSearch cluster |
| linkis.es.headers.* | None | no | Client Headers configuration |
| linkis.engineconn.concurrent.limit | 100 | no | Maximum engine concurrency of ElasticSearch cluster |

## 3. The use of ElasticSearch EngineConn
### 3.1 Ready to operate
You need to configure ElasticSearch connection information, including connection address information and user name and password (if needed).

![ElasticSearch](https://user-images.githubusercontent.com/22620332/182787195-8051bf25-1e1e-47e5-ad88-4896278857f2.png)

Figure 3-1 ElasticSearch Configuration information

You can also specify in the RuntimeMap of the submitted task.
```shell
linkis.es.cluster
linkis.es.username |
linkis.es.password
```

### 3.2 How to use Linkis SDK

Linkis provides a client method to call ElasticSearch tasks. The call method is through the SDK provided by LinkisClient. We provide java and scala two ways to call, the specific usage can refer to [JAVA SDK Manual](../user_guide/sdk-manual.md).

For the ElasticSearch task, you only need to change the EngineConnType and CodeType parameters in the Demo.

```java
Map<String, Object> labels = new HashMap<String, Object>();
labels.put(LabelKeyConstant.ENGINE_TYPE_KEY, "elasticsearch-7.6.2"); // required engineType Label
labels.put(LabelKeyConstant.USER_CREATOR_TYPE_KEY, "hadoop-IDE");// required execute user and creator
labels.put(LabelKeyConstant.CODE_TYPE_KEY, "elasticsearch"); // required codeType
```

### 3.2 How to use Linkis-cli
**-codeType parameter description**
- sql/essql:
- json/esjson:request parameters are submitted in JSON format

**Using the sample**

After Linkis 1.0, you can submit tasks through cli. We only need to specify the corresponding EngineConn and CodeType tag types. The use of ElasticSearch is as follows.
```shell
sh ./bin/linkis-cli -submitUser hadoop -engineType elasticsearch-7.6.2 -codeType json -code '{"query": {"match": {"order_id": "584677"}}}' -runtimeMap linkis.es.http.method=GET -runtimeMap linkis.es.http.endpoint=/kibana_sample_data_ecommerce/_search
```
Specific use can refer to: [Linkis CLI Manual](../user_guide/linkiscli-manual.md).


## 4. ElasticSearch EngineConn user settings

ElasticSearch user settings are mainly JDBC connection information, but it is recommended that users encrypt and manage this password and other information.
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ sidebar_position: 6
> web端是使用nginx作为静态资源服务器的,访问请求流程是:`Linkis管理台请求->nginx ip:port->linkis-gateway ip:port-> 其他服务`

Linkis 提供了单独的前端管理台功能,提供了展示历史任务的全局历史、修改用户参数、管理ECM和微服务等功能,部署前端管理台前需要先将Linkis后端进行部署,Linkis的部署手册见:[Linkis部署手册](deployment/quick-deploy.md)
Linkis 提供了单独的前端管理台功能,提供了展示历史任务的全局历史、修改用户参数、管理ECM和微服务等功能,部署前端管理台前需要先将Linkis后端进行部署,Linkis的部署手册见:[Linkis部署手册](quick-deploy.md)


## 1 准备工作
Expand Down
Loading

0 comments on commit 0a606d7

Please sign in to comment.