diff --git a/end2endTest/README.md b/end2endTest/README.md index cec453af..10c5bc5b 100644 --- a/end2endTest/README.md +++ b/end2endTest/README.md @@ -30,7 +30,7 @@ pip3 install pytest opcua pyepics pip3 install run-iocsh -i https://artifactory.esss.lu.se/artifactory/api/pypi/pypi-virtual/simple ``` -You must configure EPICS environment variables before running the test suite. +You must configure the EPICS environment before running the test suite. For the E3 environment, this requires you to ``source setE3Env.bash``. At least `EPICS_BASE` and `EPICS_HOST_ARCH` need to be set. @@ -55,7 +55,7 @@ signals are available in OPC UA namespace 2. For further information on the server configuration, see [simulation server](test/server/README.md). ### IOC -A test IOC is provided that translates the OPC UA variables from the test server. +A test IOC is provided that translates the OPC UA variables from the test server. The following records are defined: | Record | EPICS Type | OPC-UA Type | Record | EPICS Type | OPC-UA Type | @@ -86,10 +86,10 @@ cmd/ and db/ subdirectories. ## Python Test Files The pytest framework [2] is used to implement the test cases. Individual test cases are provided -as python functions (defs) in [\(opcua_test_cases.py\)](test/opcua_test_cases.py). Under the hood, +as python functions (defs) in [\(opcua_test_cases.py\)](test/opcua_test_cases.py). Under the hood, run_iocsh [3] and pyepics [4] are used for communication with the test IOC. -To add a new test case, simply add a new funtion (def) to [\(opcua_test_cases.py\)](test/opcua_test_cases.py), +To add a new test case, simply add a new funtion (def) to [\(opcua_test_cases.py\)](test/opcua_test_cases.py), ensuring that the function name begins with the prefix ``test_`` ### Test Classes and Test Cases @@ -98,54 +98,54 @@ The test points are split into four classes: #### Connection tests (TestConnectionTests) - 1. **_test_connect_disconnect_**: start and stop the test IOC 5 times. Parse the IOC output, + 1. **_test_connect_disconnect_**: start and stop the test IOC 5 times. Parse the IOC output, and check it connects and disconnects to the OPC-UA server successfully. - 2. **_test_connect_reconnect_**: Start the server, start the IOC. Stop the server, + 2. **_test_connect_reconnect_**: Start the server, start the IOC. Stop the server, check for appropriate messaging. Start the server, check that the IOC reconnects. 3. **_test_no_connection_**: Start an IOC with no server running. Check the module reports appropriately. - 4. **_test_shutdown_on_ioc_reboot_**: Start the server. Start an IOC and ensure connection - is made to the server. Shutdown the IOC and endure that the subscriptions and sessions are + 4. **_test_shutdown_on_ioc_reboot_**: Start the server. Start an IOC and ensure connection + is made to the server. Shutdown the IOC and endure that the subscriptions and sessions are cleanly disconnected. #### Variable tests (TestVariableTests) - 1. **_test_server_status_**: Check the informational values provided by the server are being + 1. **_test_server_status_**: Check the informational values provided by the server are being translated via the module. - 2. **_test_variable_pvget_**: Start the test IOC and use pvget to read the ``TstRamp`` PV + 2. **_test_variable_pvget_**: Start the test IOC and use pvget to read the ``TstRamp`` PV value multiple times (every second). Check that it is incrementing as a ramp. - 3. **_test_read_variable_**: Read the deafult value of a variable from the opcua server and - check it matches the expected value. Parametrised for all supported datatypes + 3. **_test_read_variable_**: Read the deafult value of a variable from the opcua server and + check it matches the expected value. Parametrised for all supported datatypes (boolean, sbyte, byte, int16, uint16, int32, uint32, int64, uint64, float, double, string.) - 4. **_test_write_variable_**: Write a known value to the opcua server via the output PV linked - to the variable. Read back via the input PV and check the values match. Parametrised for all + 4. **_test_write_variable_**: Write a known value to the opcua server via the output PV linked + to the variable. Read back via the input PV and check the values match. Parametrised for all supported datatypes. (boolean, sbyte, byte, int16, uint16, int32, uint32, int64, uint64, float, double, string.) - 5. **_test_timestamps_**: Start the test server in a shell session with with a fake time in - the past, using libfaketime [5]. Check that the timestamp for the PV read matches the - known fake time given to the server. If they match, the OPCUA EPICS module is correctly + 5. **_test_timestamps_**: Start the test server in a shell session with with a fake time in + the past, using libfaketime [5]. Check that the timestamp for the PV read matches the + known fake time given to the server. If they match, the OPCUA EPICS module is correctly pulling the timestamps from the OPCUA server (and not using a local timestamp). #### Performance tests (TestPerformanceTests) - 1. **_test_write_performance_**: Write 5000 variable values and measure time and memory + 1. **_test_write_performance_**: Write 5000 variable values and measure time and memory consumption before and after. Repeat 10 times - 2. **_test_read_performance_**: Read 5000 variable values and measure time and memory + 2. **_test_read_performance_**: Read 5000 variable values and measure time and memory consumption before and after. Repeat 10 times - + #### Negative tests (TestNegativeTests) - 1. **_test_no_server_**: Start an OPC-UA IOC with no server running. + 1. **_test_no_server_**: Start an OPC-UA IOC with no server running. Check the module reports this correctly. 2. **_test_bad_var_name_**: Specify an incorrect variable name in a db record. @@ -158,22 +158,22 @@ The test points are split into four classes: ### Running the test suite You can run the test suite from the root of the repository wuth the following command: ``` -pytest -v test/opcua_test_cases.py +pytest -v ``` To view the stdout output from the tests in real-time, you can provide the ``-s`` flag: ``` -pytest -v -s test/opcua_test_cases.py +pytest -v -s ``` To run all tests in a class: ``` -pytest -v test/opcua_test_cases.py::TestConnectionTests +pytest -v -k TestConnectionTests ``` To run an individual test point: ``` -pytest -v test/opcua_test_cases.py::TestConnectionTests::test_connect_disconnect +pytest -v -k test_connect_disconnect ``` diff --git a/end2endTest/cmds/test_pv.cmd b/end2endTest/cmds/test_pv.cmd index a5705f91..a34d73c6 100644 --- a/end2endTest/cmds/test_pv.cmd +++ b/end2endTest/cmds/test_pv.cmd @@ -16,4 +16,3 @@ iocshLoad("$(opcua_DIR)/opcua.iocsh", "P=OPC:,SESS=$(SESSION),SUBS=$(SUBSCRIPT), dbLoadRecords("test_pv.db", "OPCSUB=$(SUBSCRIPT), NS=$(OPCNAMESPACE)") iocInit() - diff --git a/end2endTest/cmds/test_pv_neg.cmd b/end2endTest/cmds/test_pv_neg.cmd index 50f8c9f1..70faeace 100644 --- a/end2endTest/cmds/test_pv_neg.cmd +++ b/end2endTest/cmds/test_pv_neg.cmd @@ -16,4 +16,3 @@ iocshLoad("$(opcua_DIR)/opcua.iocsh", "P=OPC:,SESS=$(SESSION),SUBS=$(SUBSCRIPT), dbLoadRecords("test_pv_neg.db", "OPCSUB=$(SUBSCRIPT), NS=$(OPCNAMESPACE)") iocInit() - diff --git a/end2endTest/opcua_test_cases.py b/end2endTest/test_opcua.py similarity index 94% rename from end2endTest/opcua_test_cases.py rename to end2endTest/test_opcua.py index 392f472e..092cc6be 100644 --- a/end2endTest/opcua_test_cases.py +++ b/end2endTest/test_opcua.py @@ -1,14 +1,15 @@ -from epics import PV -from time import sleep -from run_iocsh import IOC -from os import environ -from datetime import datetime import os -import pytest -import subprocess import resource -import time import signal +import subprocess +import time +from datetime import datetime +from os import environ +from time import sleep + +import pytest +from epics import PV, ca +from run_iocsh import IOC class opcuaTestHarness: @@ -197,6 +198,9 @@ def test_inst(): yield the harness handle to the test, close the server on test end / failure """ + # Initialize channel access + ca.initialize_libca() + # Create handle to Test Harness test_inst = opcuaTestHarness() # Poll to see if the server is running @@ -208,11 +212,16 @@ def test_inst(): test_inst.start_server() # Drop to test yield test_inst + # Shutdown server by sending terminate signal test_inst.stop_server() # Check server is stopped assert not test_inst.isServerRunning + # Shut down channel access + ca.flush_io() + ca.clear_cache() + # test fixture for use with timezone server @pytest.fixture(scope="function") @@ -222,6 +231,9 @@ def test_inst_TZ(): yield the harness handle to the test, close the server on test end / failure """ + # Initialize channel access + ca.initialize_libca() + # Create handle to Test Harness test_inst_TZ = opcuaTestHarness() # Poll to see if the server is running @@ -233,11 +245,16 @@ def test_inst_TZ(): test_inst_TZ.start_server_with_faketime() # Drop to test yield test_inst_TZ + # Shutdown server by sending terminate signal test_inst_TZ.stop_server_group() # Check server is stopped assert not test_inst_TZ.isServerRunning + # Shut down channel access + ca.flush_io() + ca.clear_cache() + class TestConnectionTests: nRuns = 5 @@ -281,8 +298,6 @@ def test_stop_and_restart_server(self, test_inst): ioc.start() assert ioc.is_running() - sleep(test_inst.sleepTime) - test_inst.stop_server() assert ioc.is_running() @@ -444,7 +459,6 @@ def test_variable_pvget(self, test_inst): with IOC( *test_inst.TestArgs, test_inst.cmd, - ioc_executable=test_inst.IOCSH_PATH, ): # PV name pvName = "TstRamp" @@ -609,6 +623,7 @@ def test_timestamps(self, test_inst_TZ): class TestPerformanceTests: + @pytest.mark.xfail("CI" in environ, reason="GitLab runner performance issues") def test_write_performance(self, test_inst): """ Write 5000 variable values and measure @@ -632,7 +647,7 @@ def test_write_performance(self, test_inst): writeperrun = 5000 # Run test 10 times - for j in range(testruns): + for j in range(1, testruns): # Get time and memory conspumtion before test r0 = resource.getrusage(resource.RUSAGE_THREAD) @@ -645,10 +660,7 @@ def test_write_performance(self, test_inst): # Get delta time and delta memory dt = time.perf_counter() - t0 r1 = resource.getrusage(resource.RUSAGE_THREAD) - dr = ( - resource.getrusage(resource.RUSAGE_THREAD).ru_maxrss - - r0.ru_maxrss # NoQA: E501 - ) + dr = r1.ru_maxrss - r0.ru_maxrss # NoQA: E501 # Collect data for statistics if dt > maxt: @@ -661,6 +673,7 @@ def test_write_performance(self, test_inst): print("Memory incr: ", dr) print(" before: ", r0.ru_maxrss) print(" after: ", r1.ru_maxrss) + avgt = tott / testruns print("Max time: ", "{:.3f} s".format(maxt)) @@ -669,9 +682,9 @@ def test_write_performance(self, test_inst): print("Total memory increase: ", totr) assert maxt < 17 - assert mint > 1 - assert avgt < 12 - assert totr < 1000 + assert mint > 0.8 + assert avgt < 5 + assert totr < 3000 def test_read_performance(self, test_inst): """ @@ -696,7 +709,7 @@ def test_read_performance(self, test_inst): writeperrun = 5000 # Run test 10 times - for j in range(testruns): + for j in range(1, testruns): # Get time and memory conspumtion before test r0 = resource.getrusage(resource.RUSAGE_SELF) @@ -709,10 +722,7 @@ def test_read_performance(self, test_inst): # Get delta time and delta memory dt = time.perf_counter() - t0 r1 = resource.getrusage(resource.RUSAGE_SELF) - dr = ( - resource.getrusage(resource.RUSAGE_SELF).ru_maxrss - - r0.ru_maxrss # NoQA: E501 - ) + dr = r1.ru_maxrss - r0.ru_maxrss # NoQA: E501 # Collect data for statistics if dt > maxt: @@ -725,6 +735,7 @@ def test_read_performance(self, test_inst): print("Memory incr: ", dr) print(" before: ", r0.ru_maxrss) print(" after: ", r1.ru_maxrss) + avgt = tott / testruns print("Max time: ", "{:.3f} s".format(maxt)) @@ -734,11 +745,39 @@ def test_read_performance(self, test_inst): assert maxt < 10 assert mint > 0.01 - assert avgt < 5 + assert avgt < 7 assert totr < 1000 class TestNegativeTests: + def test_no_server(self, test_inst): + """ + Start an OPC-UA IOC with no server running. + Check the module reports this correctly. + """ + + ioc = test_inst.IOC + + # Stop the running server + test_inst.stop_server() + + # Start the IOC + ioc.start() + assert ioc.is_running() + + # Check that PVs have SEVR INVALID (=3) + pv = PV("VarCheckSByte") + pv.get(timeout=test_inst.getTimeout) + assert pv.severity == 3 + + # Stop IOC, and check output + ioc.exit() + assert not ioc.is_running() + + ioc.check_output() + output = ioc.outs + print(output) + def test_bad_var_name(self, test_inst): """ Specify an incorrect variable name in a db record. @@ -753,9 +792,6 @@ def test_bad_var_name(self, test_inst): ioc.start() assert ioc.is_running() - # Wait some time - sleep(1) - # Stop IOC, and check output ioc.exit() assert not ioc.is_running() @@ -782,9 +818,6 @@ def test_wrong_datatype(self, test_inst): ioc.start() assert ioc.is_running() - # Wait some time - sleep(1) - # Stop IOC, and check output ioc.exit() assert not ioc.is_running()