Skip to content

Commit

Permalink
Updated the integ-test docker README file
Browse files Browse the repository at this point in the history
Signed-off-by: Norman Jordan <[email protected]>
  • Loading branch information
normanj-bitquill committed Jan 16, 2025
1 parent 5d40352 commit a570e67
Showing 1 changed file with 24 additions and 1 deletion.
25 changes: 24 additions & 1 deletion docs/docker/integ-test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ the docker image.
Spark Connect is also running in this container and can be used to easily issue queries to run. The port for
Spark Connect is exposed to the host OS.

Spark is configured to have an OpenSearch datastore with the catalog name `dev`. Indices on the OpenSearch
server can be queries as `dev.default.<INDEX_NAME>`.

### Spark Worker

The Spark worker node. It is configured to use an external Hive metastore in the container `metastore`. The
Expand Down Expand Up @@ -68,7 +71,7 @@ It will contain the S3 tables data.

This container also has a docker volume used to persist the S3 data.

### Configuration-Updated
### Configuration-Updater

A temporary container that is used to configure the OpenSearch and Minio containers. It is run after both
of those have started up. For Minio, it will add the `integ-test` bucket and create an access key. For
Expand Down Expand Up @@ -109,6 +112,26 @@ spark.sql("CREATE EXTERNAL TABLE foo (id int, name varchar(100)) location 's3a:/
spark.sql("INSERT INTO foo (id, name) VALUES(1, 'Foo')")
```

A REST call to the OpenSearch container can be used to query the table using the Async API.
```shell
curl \
-u 'admin:C0rrecthorsebatterystaple.' \
-X POST \
-H 'Content-Type: application/json' \
-d '{"datasource": "mys3", "lang": "sql", "query": "SELECT * FROM mys3.default.foo"}' \
http://localhost:9200/_plugins/_async_query
```

When the query is finished, the results can be retrieved with a REST call to the OpenSearch container.
```shell
curl \
-u 'admin:C0rrecthorsebatterystaple.' \
-X POST \
-H 'Content-Type: application/json' \
-d '{}' \
'http://localhost:9200/query_execution_result_mys3/_search?pretty'
```

## Configuration of the Cluster

There are several settings that can be adjusted for the cluster.
Expand Down

0 comments on commit a570e67

Please sign in to comment.