Scripts to automate CLI packager and runner

tests/automation/README.md

@@ -0,0 +1,150 @@
+# Holoscan CLI Test Automation
+
+This directory contains scripts to automate the testing of packaging and running applications designed using [Holoscan SDK](https://developer.nvidia.com/holoscan-sdk).
+
+## Requirements
+
+To successfully execute the test automation script, the system must meet the requirements specified by both the [Holoscan SDK](https://docs.nvidia.com/holoscan/sdk-user-guide/sdk_installation.html) and the [Holoscan CLI](https://github.com/nvidia-holoscan/holoscan-cli). Additionally, the following tools are required to package and run the application:
+
+* Holoscan CLI
+* jq
+* curl
+* An [NGC](https://catalog.ngc.nvidia.com/) account, which must be logged in via `docker login nvcr.io`
+
+## Usage
+
+To run the test automation script, navigate to the automation directory, then execute the script with the application directory as an argument:
+
+```
+cd tests/automation
+./test.sh [application-directory]
+# for example
+./tests.sh ./endoscopy_tool_tracking_cpp
+```
+
+The `test.sh` script invokes `check.sh` to verify that all system requirements are met. It will exit if any of the requirements are missing.
+
+The automation script performs the following steps: 
+
+1. Clones the specified git repository. 
+2. Packages the application. 
+3. Downloads the configured test data. 
+4. Run the packaged application. 
+5. Cleans up any downloaded and generated data.
+
+### Adding New Application
+
+Automating the testing of a new Holoscan-enabled application is straightforward and can be accomplished with the following steps:
+
+1. Create a new directory and include a `config.json` file that defines the application and specifies where the script can retrieve the test data. Refer to the sections below for examples and the configuration schema. 
+2. Optionally, copy the application's configuration YAML file and modify the values to ensure that the application can start and exit on its own.
+
+This is all that's required.
+
+### Sample Configuration 
+
+#### Holoscan Application
+
+Here’s a sample configuration for testing the C++ version of the Video Replayer:
+
+With this configuration, the test automation script will clone the [Holoscan SDK repository](https://github.com/nvidia-holoscan/holoscan-sdk) from GitHub, download test data from NGC, package the application with `--include holoviz`, and run the application using the supplied `app.yaml` configuration file along with the `--render` option.
+
+```json
+{
+  "name": "video-replayer-cpp",
+  "source": {
+    "repo": "https://github.com/nvidia-holoscan/holoscan-sdk.git",
+    "path": "examples/video_replayer/cpp",
+    "lang": "cpp"
+ },
+  "config": {
+    "source": "local",
+    "path": "./app.yaml"
+ },
+  "package": {
+    "args": "--includes holoviz"
+ },
+  "run": {
+    "args": "--render"
+ },
+  "data": {
+    "dirname" : "racerx",
+    "source" : "https://api.ngc.nvidia.com/v2/resources/org/nvidia/team/clara-holoscan/holoscan_racerx_video/20231009/files"
+ }
+}
+```
+
+#### Holohub Application
+
+Another sample configuration is used for testing the Endoscopy Tool Tracking application from Holohub:
+
+In this case, the test automation script will clone the [Holohub](https://github.com/nvidia-holoscan/holohub) repository, invoke the `devcontainer` script to build the specified application, utilize test data downloaded by the `devcontainer` script, and package the application using the provided `app.yaml` file with additional arguments defined in `package.args`. Finally, it executes the application with the arguments specified in `run.args`.
+
+
+```json
+{
+  "name": "endoscopy-tool-tracking-cpp",
+  "source": {
+    "repo": "https://github.com/nvidia-holoscan/holohub.git",
+    "app": "endoscopy_tool_tracking",
+    "lang": "python",
+    "path": "applications/endoscopy_tool_tracking/python/endoscopy_tool_tracking.py"
+ },
+  "config": {
+    "source": "local",
+    "path": "./app.yaml"
+ },
+  "package": {
+    "args": "--includes onnx holoviz --add <src>/install/lib --add <src>/install/python/lib/"
+ },
+  "run": {
+    "args": "--render"
+ },
+  "data": {
+    "source" : "local-holohub",
+    "dirname" : "endoscopy"
+ }
+}
+```
+
+> [!NOTE]
+> The test automation script supports packaging Holohub applications from the `install/` generated by the devcontainer script. If a desired application does not create artifacts in the `install/` directory, update the associated CMake configuration to [install](https://cmake.org/cmake/help/latest/command/install.html) the application first.
+
+
+### Configuration Schema
+
+```
+{
+ "name": "<name of the application>", # Required
+ "source": {
+ "repo": "<HTTP-based git url>.git", # Required
+ "path": "<application's relative path from the root of the repository", # Required
+ "lang": "<cpp|python>", # Required
+ "app": "Holohub only: name of application. Same as the application name used by the devcontainer script."
+ },
+ "config": {
+ "source": "[local]", # optional value
+ "path": "./app.yaml" # Required
+ },
+ "package": {
+ "args": "<additional arguments to pass to the CLI Packager>"
+ },
+ "run": {
+ "args": "<additional arguments to pass to the CLI Runner>"
+ },
+ "data": {
+ "source" : "<NGC URL|local-holohub>", # 'local-holohub' means to use test data downloaded by the devcontainer script
+ "dirname" : "<directory name to store the test data. For Holohub, the relative path to the data directory from Holohub.>"
+ }
+}
+```
+
+
+## Links
+
+- [Holoscan SDK User Guide](https://docs.nvidia.com/holoscan/sdk-user-guide)
+- [Holoscan Application Package Specification (HAP)](https://docs.nvidia.com/holoscan/sdk-user-guide/cli/hap.html)
+- [Holscan SDK Github Repository](https://github.com/nvidia-holoscan/holoscan-sdk)
+- [Holohub Github Repository](https://github.com/nvidia-holoscan/holohub/)
+- [Holoscan CLI Github Repository](https://github.com/nvidia-holoscan/holoscan-cli)
+- [NGC](https://ngc.nvidia.com/)

tests/automation/artifacts.json

@@ -0,0 +1,32 @@
+{
+  "2.9.0": {
+    "holoscan": {
+      "debian-version": "2.9.0",
+      "wheel-version": "2.9.0",
+      "base-images": {
+        "dgpu": "nvcr.io/nvidia/cuda:12.6.0-runtime-ubuntu22.04",
+        "igpu": "nvcr.io/nvidia/tensorrt:24.08-py3-igpu"
+      },
+      "build-images": {
+        "igpu": {
+          "jetson-agx-orin-devkit": "nvcr.io/nvstaging/holoscan/holoscan:v2.9.0.1-igpu",
+          "igx-orin-devkit": "nvcr.io/nvstaging/holoscan/holoscan:v2.9.0.1-igpu",
+          "sbsa": "nvcr.io/nvstaging/holoscan/holoscan:v2.9.0.1-igpu"
+        },
+        "dgpu": {
+          "x64-workstation": "nvcr.io/nvstaging/holoscan/holoscan:v2.9.0.1-dgpu",
+          "igx-orin-devkit": "nvcr.io/nvstaging/holoscan/holoscan:v2.9.0.1-dgpu",
+          "sbsa": "nvcr.io/nvstaging/holoscan/holoscan:v2.9.0.1-dgpu",
+          "clara-agx-devkit": "nvcr.io/nvstaging/holoscan/holoscan:v2.9.0.1-dgpu"
+        },
+        "cpu": {
+          "x64-workstation": "nvcr.io/nvstaging/holoscan/holoscan:v2.9.0.1-dgpu"
+        }
+      }
+    },
+    "health-probes": {
+      "linux/amd64": "https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.19/grpc_health_probe-linux-amd64",
+      "linux/arm64": "https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.19/grpc_health_probe-linux-arm64"
+    }
+  }
+}

tests/automation/check.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#===============================================================================
+
+echo "Checking system requirements..."
+
+status=0
+if ! command -v curl 2>&1 >/dev/null
+then
+    echo "curl could not be found, use the following command to install curl"
+    echo "$ sudo apt install curl"
+    status=-1
+fi
+
+if ! command -v jq 2>&1 >/dev/null
+then
+    echo "jq could not be found, use the following command to install jq"
+    echo "$ sudo apt install jq"
+    status=-1
+fi
+
+if ! command -v holoscan 2>&1 >/dev/null
+then
+    echo "Holoscan CLI could not be found, use the following command to install holoscan"
+    echo "$ pip install holoscan-cli"
+    status=-1
+fi
+
+if ! docker login nvcr.io < /dev/null >& /dev/null
+then
+    echo "Please login to nvcr.io. For example:"
+    echo "$ docker login nvcr.io"
+    status=-1
+fi
+
+exit $status