Add README and tidy up

Signed-off-by: Victor Chang <[email protected]>

tests/automation/README.md

@@ -0,0 +1,136 @@
+# Holoscan CLI Test Automation
+
+This directory contains scripts to automate the testing of packaging and running applications designed using [Holoscan SDK](https://developer.nvidia.com/holoscan-sdk).
+
+## Requirements
+
+To successfully execute the test automation script, the system must meet the requirements specified by both the [Holoscan SDK](https://docs.nvidia.com/holoscan/sdk-user-guide/sdk_installation.html) and the [Holoscan CLI](https://github.com/nvidia-holoscan/holoscan-cli). Additionally, the following tools are required to package and run the application:
+
+* Holoscan CLI
+* jq
+* curl
+* An [NGC](https://catalog.ngc.nvidia.com/) account, which must be logged in via `docker login nvcr.io`
+
+## Usage
+
+To run the test automation script, navigate to the automation directory, then execute the script with the application directory as an argument:
+
+```
+cd tests/automation
+./test.sh [application-directory]
+# for example
+./tests.sh ./endoscopy_tool_tracking_cpp
+```
+
+The `test.sh` script invokes `check.sh` to verify that all system requirements are met. It will exit if any of the requirements are missing.
+
+The automation script performs the following steps: 
+
+1. Clones the specified git repository. 
+2. Packages the application. 
+3. Downloads the configured test data. 
+4. Run the packaged application. 
+5. Cleans up any downloaded and generated data.
+
+### Adding New Application
+
+Automating the testing of a new Holoscan-enabled application is straightforward and can be accomplished with the following steps:
+
+1. Create a new directory and include a `config.json` file that defines the application and specifies where the script can retrieve the test data. Refer to the sections below for examples and the configuration schema. 
+2. Optionally, copy the application's configuration YAML file and modify the values to ensure that the application can start and exit on its own.
+
+This is all that's required.
+
+### Sample Configuration 
+
+#### Holoscan Application
+
+Here’s a sample configuration for testing the C++ version of the Video Replayer:
+
+With this configuration, the test automation script will clone the [Holoscan SDK repository](https://github.com/nvidia-holoscan/holoscan-sdk) from GitHub, download test data from NGC, package the application with `--include holoviz`, and run the application using the supplied `app.yaml` configuration file along with the `--render` option.
+
+```json
+{
+  "name": "video-replayer-cpp",
+  "source": {
+    "repo": "https://github.com/nvidia-holoscan/holoscan-sdk.git",
+    "path": "examples/video_replayer/cpp",
+    "lang": "cpp"
+ },
+  "config": {
+    "source": "local",
+    "path": "./app.yaml"
+ },
+  "package": {
+    "args": "--includes holoviz"
+ },
+  "run": {
+    "args": "--render"
+ },
+  "data": {
+    "dirname" : "racerx",
+    "source" : "https://api.ngc.nvidia.com/v2/resources/org/nvidia/team/clara-holoscan/holoscan_racerx_video/20231009/files"
+ }
+}
+```
+
+#### Holohub Application
+
+Another sample configuration is used for testing the Endoscopy Tool Tracking application from Holohub:
+
+In this case, the test automation script will clone the [Holohub](https://github.com/nvidia-holoscan/holohub) repository, invoke the `devcontainer` script to build the specified application, utilize test data downloaded by the `devcontainer` script, and package the application using the provided `app.yaml` file with additional arguments defined in `package.args`. Finally, it executes the application with the arguments specified in `run.args`.
+
+
+```json
+{
+  "name": "endoscopy-tool-tracking-cpp",
+  "source": {
+    "repo": "https://github.com/nvidia-holoscan/holohub.git",
+    "app": "endoscopy_tool_tracking",
+    "lang": "python",
+    "path": "applications/endoscopy_tool_tracking/python/endoscopy_tool_tracking.py"
+ },
+  "config": {
+    "source": "local",
+    "path": "./app.yaml"
+ },
+  "package": {
+    "args": "--includes onnx holoviz --add <src>/install/lib --add <src>/install/python/lib/"
+ },
+  "run": {
+    "args": "--render"
+ },
+  "data": {
+    "source" : "local-holohub",
+    "dirname" : "endoscopy"
+ }
+}
+```
+
+### Configuration Schema
+
+```
+{
+ "name": "<name of the application>", # Required
+ "source": {
+ "repo": "<HTTP-based git url>.git", # Required
+ "path": "<application's relative path from the root of the repository", # Required
+ "lang": "<cpp|python>", # Required
+ "app": "Holohub only: name of application. Same as the application name used by the devcontainer script."
+ },
+ "config": {
+ "source": "[local]", # optional value
+ "path": "./app.yaml" # Required
+ },
+ "package": {
+ "args": "<additional arguments to pass to the CLI Packager>"
+ },
+ "run": {
+ "args": "<additional arguments to pass to the CLI Runner>"
+ },
+ "data": {
+ "source" : "<NGC URL|local-holohub>", # 'local-holohub' means to use test data downloaded by the devcontainer script
+ "dirname" : "<directory name to store the test data. For Holohub, the relative path to the data directory from Holohub.>"
+ }
+}
+```

tests/automation/check.sh

@@ -17,37 +17,34 @@
 #===============================================================================
 
 echo "Checking system requirements..."
-if ! command -v unzip 2>&1 >/dev/null
-then
-    echo "unzip could not be found, installing unzip"
-    echo "$ sudo apt install unzip"
-    exit -1
-fi
 
+status=0
 if ! command -v curl 2>&1 >/dev/null
 then
-    echo "jq could not be found, installing curl"
+    echo "jq could not be found, use the following command to install curl"
     echo "$ sudo apt install curl"
-    exit -1
+    status=-1
 fi
 
 if ! command -v jq 2>&1 >/dev/null
 then
-    echo "jq could not be found, installing jq"
+    echo "jq could not be found, use the following command to install jq"
     echo "$ sudo apt install jq"
-    exit -1
+    status=-1
 fi
 
 if ! command -v holoscan 2>&1 >/dev/null
 then
-    echo "Holoscan CLI could not be found, installing holoscan"
+    echo "Holoscan CLI could not be found, use the following command to install holoscan"
     echo "$ pip install holoscan-cli"
-    exit -1
+    status=-1
 fi
 
 if ! docker login nvcr.io < /dev/null >& /dev/null
 then
     echo "Please login to nvcr.io. For example:"
     echo "$ docker login nvcr.io"
-    exit -1
-fi
+    status=-1
+fi
+
+exit $status