From a570e67c4339a65e189fa46641763bb8e38925c9 Mon Sep 17 00:00:00 2001 From: Norman Jordan Date: Thu, 16 Jan 2025 09:24:28 -0800 Subject: [PATCH] Updated the integ-test docker README file Signed-off-by: Norman Jordan --- docs/docker/integ-test/README.md | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/docs/docker/integ-test/README.md b/docs/docker/integ-test/README.md index 47b303abb..e85c24e50 100644 --- a/docs/docker/integ-test/README.md +++ b/docs/docker/integ-test/README.md @@ -39,6 +39,9 @@ the docker image. Spark Connect is also running in this container and can be used to easily issue queries to run. The port for Spark Connect is exposed to the host OS. +Spark is configured to have an OpenSearch datastore with the catalog name `dev`. Indices on the OpenSearch +server can be queries as `dev.default.`. + ### Spark Worker The Spark worker node. It is configured to use an external Hive metastore in the container `metastore`. The @@ -68,7 +71,7 @@ It will contain the S3 tables data. This container also has a docker volume used to persist the S3 data. -### Configuration-Updated +### Configuration-Updater A temporary container that is used to configure the OpenSearch and Minio containers. It is run after both of those have started up. For Minio, it will add the `integ-test` bucket and create an access key. For @@ -109,6 +112,26 @@ spark.sql("CREATE EXTERNAL TABLE foo (id int, name varchar(100)) location 's3a:/ spark.sql("INSERT INTO foo (id, name) VALUES(1, 'Foo')") ``` +A REST call to the OpenSearch container can be used to query the table using the Async API. +```shell +curl \ + -u 'admin:C0rrecthorsebatterystaple.' \ + -X POST \ + -H 'Content-Type: application/json' \ + -d '{"datasource": "mys3", "lang": "sql", "query": "SELECT * FROM mys3.default.foo"}' \ + http://localhost:9200/_plugins/_async_query +``` + +When the query is finished, the results can be retrieved with a REST call to the OpenSearch container. +```shell +curl \ + -u 'admin:C0rrecthorsebatterystaple.' \ + -X POST \ + -H 'Content-Type: application/json' \ + -d '{}' \ + 'http://localhost:9200/query_execution_result_mys3/_search?pretty' +``` + ## Configuration of the Cluster There are several settings that can be adjusted for the cluster.