diff --git a/.gitignore b/.gitignore index 03940d33..795161d9 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ *.tgz *.tar.gz osc +ibs +test.log diff --git a/SAPHanaSR-angi.spec b/SAPHanaSR-angi.spec index 8d7a8be6..702be97e 100644 --- a/SAPHanaSR-angi.spec +++ b/SAPHanaSR-angi.spec @@ -21,7 +21,7 @@ License: GPL-2.0 Group: Productivity/Clustering/HA AutoReqProv: on Summary: Resource agents to control the HANA database in system replication setup -Version: 1.2.3 +Version: 1.2.4 Release: 0 Url: https://www.suse.com/c/fail-safe-operation-of-sap-hana-suse-extends-its-high-availability-solution/ diff --git a/SAPHanaSR-tester.spec b/SAPHanaSR-tester.spec index fb7094de..03e49b61 100644 --- a/SAPHanaSR-tester.spec +++ b/SAPHanaSR-tester.spec @@ -19,7 +19,7 @@ License: GPL-2.0 Group: Productivity/Clustering/HA AutoReqProv: on Summary: Test suite for SAPHanaSR clusters -Version: 1.2.7 +Version: 1.2.8 Release: 0 Url: https://www.suse.com/c/fail-safe-operation-of-sap-hana-suse-extends-its-high-availability-solution/ diff --git a/man-tester/SAPHanaSR-testCluster.8 b/man-tester/SAPHanaSR-testCluster.8 index 553a2bb7..8762f45c 100644 --- a/man-tester/SAPHanaSR-testCluster.8 +++ b/man-tester/SAPHanaSR-testCluster.8 @@ -3,7 +3,7 @@ .TH SAPHanaSR-testCluster 8 "20 Nov 2023" "" "SAPHanaSR-angi" .\" .SH NAME -SAPHanaSR-testCluster \- Run functional testing for SAPHanaSR clusters. +SAPHanaSR-testCluster \- Run functional tests for SAPHanaSR clusters. .PP .\" .SH SYNOPSIS @@ -78,7 +78,8 @@ Usage, syntax or execution errors. The functional test "nop" is performed on the Linux cluster defined in properties_q42.json, nodes are node1 and node2. A dedicated working directory and logfile for this test is used. -See also manual page SAPHanaSR-tester(7) and SAPHanaSR-tests-syntax(7). +See also manual page SAPHanaSR-tester(7), SAPHanaSR-tests-syntax(5), +SAPHanaSR-tests-angi-ScaleUp(7) and SAPHanaSR-tests-description(7). .PP .RS 2 # mkdir ~/test_nop; cd ~/test_nop @@ -148,7 +149,9 @@ TODO .\" .SH REQUIREMENTS .\" -See the REQUIREMENTS section in SAPHanaSR-tester(7). +See the REQUIREMENTS section in SAPHanaSR-tester(7) and SAPHanaSR-angi(7). +Of course, HANA database and Linux cluster have certain requirements. +Please refer to the product documentation. .PP .\" .SH BUGS @@ -158,7 +161,8 @@ Please report any other feedback and suggestions to feedback@suse.com. .PP .\" .SH SEE ALSO -\fBSAPHanaSR-tester\fP(7) , \fBSAPHanaSR-tests-syntax\fP(7) , +\fBSAPHanaSR-tester\fP(7) , \fBSAPHanaSR-tests-syntax\fP(5) , +\fBSAPHanaSR-tests-description\fP(7) , \fBSAPHanaSR-tests-angi-ScaleUp\fP(7) , \fBSAPHanaSR-tests-angi-ScaleOut\fP(7) , \fBSAPHanaSR-angi\fP(7) , \fBSAPHanaSR-showAttr\fP(8) , \fBcrm_mon\fP(8) , \fBcrm\fP(8) , \fBcs_clusterstate\fP(8) @@ -169,7 +173,7 @@ F.Herschel, L.Pinne. .PP .\" .SH COPYRIGHT -(c) 2023 SUSE Linux GmbH, Germany. +(c) 2023 SUSE LLC .br The package SAPHanaSR-tester comes with ABSOLUTELY NO WARRANTY. .br diff --git a/man-tester/SAPHanaSR-tester.7 b/man-tester/SAPHanaSR-tester.7 index dd0fbdb7..9cb443d3 100644 --- a/man-tester/SAPHanaSR-tester.7 +++ b/man-tester/SAPHanaSR-tester.7 @@ -63,14 +63,14 @@ See manual page SAPHanaSR-tests-syntax(5) for details, see also example below. \fB*\fP Processing: .\" TODO SAPHanaSR-testCluster -.\" TODO shell scripts shiped with SAPhanaSR-tester, e.g. test_ +.\" TODO shell scripts shiped with SAPhanaSR-tester, e.g. sct_test_ .\" TODO custom scripts and test automation frameworks -See manual page SAPHanaSR-testCluster(8). +See manual page SAPHanaSR-testCluster(8) and SAPHanaSR-tests-description(7). \fB*\fP Output: Test results .\" TODO output of SAPHanaSR-testCluster -See manual page SAPHanaSR-testCluster(8). +See manual page SAPHanaSR-testCluster(8) and SAPHanaSR-tests-description(7). .PP .\" .SH EXAMPLES @@ -146,8 +146,9 @@ Note: Never do this on production systems. \fB*\fR Showing predefined functional scale-up test cases. .PP The predefined functional tests for the SAPHanaSR-angi scale-up scenario are -shown. See manual page SAPHanaSR-tests-angi-ScaleUp(7) and -SAPHanaSR-testCluster(8) for how to run this tests. +shown. See also manual page SAPHanaSR-tests-angi-ScaleUp(7). See manual page +SAPHanaSR-tests-description(7) and SAPHanaSR-testCluster(8) for how to run this +tests. .PP .RS 2 # ls /usr/share/SAPHanaSR-tester/json/angi-ScaleUp/*.json | \\ @@ -236,6 +237,7 @@ Please report any other feedback and suggestions to feedback@suse.com. .\" .SH SEE ALSO \fBSAPHanaSR-testCluster\fP(8) , \fBSAPHanaSR-tests-syntax\fP(5) , +\fBSAPHanaSR-tests-description\fP(7) , \fBSAPHanaSR-tests-angi-ScaleUp\fP(7) , \fBSAPHanaSR-tests-angi-ScaleOut\fP(7) , \fBSAPHanaSR-angi\fP(7) , \fBSAPHanaSR-showAttr\fP(8) , \fBcrm_mon\fP(8) , \fBssh-keygen\fP(1) , \fBssh-copy-id\fP(1) , @@ -252,7 +254,7 @@ F.Herschel, L.Pinne. .PP .\" .SH COPYRIGHT -(c) 2023 SUSE Linux GmbH, Germany. +(c) 2023 SUSE LLC .br The package SAPHanaSR-tester comes with ABSOLUTELY NO WARRANTY. .br diff --git a/man-tester/SAPHanaSR-tests-angi-ScaleOut.7 b/man-tester/SAPHanaSR-tests-angi-ScaleOut.7 index fea9965b..1cf36f20 100644 --- a/man-tester/SAPHanaSR-tests-angi-ScaleOut.7 +++ b/man-tester/SAPHanaSR-tests-angi-ScaleOut.7 @@ -1,6 +1,6 @@ .\" Version: 1.001 .\" -.TH SAPHanaSR-tests-angi-ScaleOut 7 "20 Nov 2023" "" "SAPHanaSR-angi" +.TH SAPHanaSR-tests-angi-ScaleOut 7 "20 Dec 2023" "" "SAPHanaSR-angi" .\" .SH NAME SAPHanaSR-tests-angi-ScaleOut \- Functional tests for SAPHanaSR Scale-Out. @@ -8,101 +8,123 @@ SAPHanaSR-tests-angi-ScaleOut \- Functional tests for SAPHanaSR Scale-Out. .\" .SH DESCRIPTION .PP -Functional test are shipped for scale-out scenarios. This tests could be run -out-of-the-box. The test cases are defined in dedicated files. -See manual page SAPHanaSR-tests-syntax(5) for syntax details. +Functional test are shipped for the scale-out ERP scenario. This tests could +be run out-of-the-box. The test cases are defined in dedicated files. +See manual page SAPHanaSR-tests-syntax(5) for syntax details. Details like +performed steps or expected behaviour of cluster and HANA are explained in +SAPHanaSR-tests-description(7). -Entry point for all predefined tests is a clean and idle Linux cluster and a -clean HANA pair in sync. Same is true for the final state. -See manual page SAPHanaSR_maintenance_examples(7) for detecting the correct -status and watching changes near real-time. +Each test can be executed by running the command SAPHanaSR-testCluster with +appropriate parameters. See manual page SAPHanaSR-testCluster(8). .PP -Predefined functional tests for scale-out overwiev +Predefined functional tests for scale-out ERP overwiev: .TP -block_manual_takeover -blocked manual takeover, for susTkOver.py +\fBblock_manual_takeover\fP +Blocked manual takeover, for susTkOver.py. .TP -block_sr_and_freeze_prim_master_nfs -block HANA SR and freeze HANA NFS on primary master node +\fBblock_sr_and_freeze_prim_master_nfs\fP +Block HANA SR and freeze HANA NFS on primary master node (not yet implemented). .TP -block_sr_and_freeze_prim_site_nfs -block HANA SR and freeze HANA NFS on primary site +\fBblock_sr_and_freeze_prim_site_nfs\fP +Block HANA SR and freeze HANA NFS on primary site (not yet implemented). .TP -free_log_area -free HANA log area on primary site +\fBfree_log_area\fP +Free HANA log area on primary site. .TP -freeze_prim_master_nfs -freeze HANA NFS on primary master node +\fBfreeze_prim_master_nfs\fP +Freeze HANA NFS on primary master node. .TP -freeze_prim_site_nfs -freeze HANA NFS on primary site +\fBfreeze_prim_site_nfs\fP +Freeze HANA NFS on primary site. .TP -kill_prim_indexserver -Kill primary master indexserver, for susChkSrv.py +\fBfreeze_secn_site_nfs\fP +Freeze HANA NFS on secondary site (not yet implemented). .TP -kill_prim_inst -Kill primary master instance +\fBkill_prim_indexserver\fP +Kill primary master indexserver, for susChkSrv.py. .TP -kill_prim_node -Kill primary master node +\fBkill_prim_inst\fP +Kill primary master instance. .TP -kill_prim_worker_indexserver -Kill primary worker indexserver, for susChkSrv.py +\fBkill_prim_node\fP +Kill primary master node. .TP -kill_prim_worker_inst -Kill primary worker instance +\fBkill_prim_site\fP +Kill secondary site nodes (not yet implemented). .TP -kill_prim_worker_node -Kill primary worker node +\fBkill_prim_worker_indexserver\fP +Kill primary worker indexserver, for susChkSrv.py. .TP -kill_secn_indexserver -Kill secondary master indexserver, for susChkSrv.py +\fBkill_prim_worker_inst\fP +Kill primary worker instance. .TP -kill_secn_inst -Kill secondary master instance +\fBkill_prim_worker_node\fP +Kill primary worker node. .TP -kill_secn_node -Kill secondary master node +\fBkill_secn_indexserver\fP +Kill secondary master indexserver, for susChkSrv.py. .TP -kill_secn_worker_inst -Kill secondary worker instance +\fBkill_secn_inst\fP +Kill secondary master instance. .TP -kill_secn_worker_node -Kill secondary worker node +\fBkill_secn_node\fP +Kill secondary master node. .TP -maintenance_cluster_turn_hana -maintenance procedure, manually turning HANA sites +\fBkill_secn_worker_inst\fP +Kill secondary worker instance. .TP -maintenance_with_standby_nodes -standby+online secondary then standby+online primary +\fBkill_secn_worker_node\fP +Kill secondary worker node. .TP -nop -no operation - check, wait and check again (stability check) +\fBkill_secn_site\fP +Kill secondary site nodes (not yet implemented). .TP -restart_cluster - +\fBmaintenance_cluster_turn_hana\fP +Maintenance procedure, manually turning HANA sites. .TP -restart_cluster_hana_running - +\fBmaintenance_with_standby_nodes\fP +Maintenance procedure, standby+online secondary then standby+online primary (not yet implemented). .TP -restart_cluster_turn_hana - +\fBnop\fP +No operation - check, wait and check again (stability check). +.TP +\fBregister_prim_cold_hana\fP +Stop cluster, do manual takeover, leave former primary down and unregistered, start cluster +(not yet implemented). +.TP +\fBrestart_cluster\fP +Stop and restart cluster and HANA +.TP +\fBrestart_cluster_hana_running\fP +Stop and restart cluster, keep HANA running. .TP -standby_primary_node TODO -standby primary node (and online again) +\fBrestart_cluster_turn_hana\fP +Stop cluster and HANA, takeover HANA, start cluster. .TP -standby_secondary_node TODO -standby secondary node (and online again) +\fBstandby_prim_node\fP +Standby primary master node and online again. +.TP +\fBstandby_secn_node\fP +Standby secondary master node and online again. +.TP +\fBstandby_secn_worker_node\fP +Standby secondary worker node and online again. .RE .PP .\" .SH EXAMPLES .PP +* List tests for SAPHanaSR-angi scale-out ERP scenarios +.PP +.RS 2 +# ls /usr/share/SAPHanaSR-tester/json/angi-ScaleOut/ +.RE +.PP .\" .SH FILES .TP -/usr/share/SAPHanaSR-tester/json/angi-ScaleOu/ -functional tests for SAPHanaSR-angi scale-out scenarios. +/usr/share/SAPHanaSR-tester/json/angi-ScaleOut/ +functional tests for SAPHanaSR-angi scale-out ERP scenarios. .TP /usr/bin/sct_test_* shell scripts for un-easy tasks on the cluster nodes. @@ -110,7 +132,9 @@ shell scripts for un-easy tasks on the cluster nodes. .\" .SH REQUIREMENTS .PP -See the REQUIREMENTS section in SAPHanaSR-tester(7). +See the REQUIREMENTS section in SAPHanaSR-tester(7) and SAPHanaSR-angi(7). +Of course, HANA database and Linux cluster have certain requirements. +Please refer to the product documentation. .\" .SH BUGS In case of any problem, please use your favourite SAP support process to open @@ -120,7 +144,8 @@ Please report any other feedback and suggestions to feedback@suse.com. .\" .SH SEE ALSO \fBSAPHanaSR-tester\fP(7) , \fBSAPHanaSR-testCluster\fP(8) , -\fBSAPHanaSR-tests-angi-ScaleUp\fP(7) , \fBSAPHanaSR-tests-syntax\fP(5) , +\fBSAPHanaSR-tests-description\fP(7) , \fBSAPHanaSR-tests-syntax\fP(5) , +\fBSAPHanaSR-tests-angi-ScaleUp\fP(7) , \fBSAPHanaSR-angi\fP(7) , \fBSAPHanaSR-showAttr\fP(8) .PP .\" @@ -129,7 +154,7 @@ F.Herschel, L.Pinne. .PP .\" .SH COPYRIGHT -(c) 2023 SUSE Linux GmbH, Germany. +(c) 2023 SUSE LLC .br The package SAPHanaSR-tester comes with ABSOLUTELY NO WARRANTY. .br diff --git a/man-tester/SAPHanaSR-tests-angi-ScaleUp.7 b/man-tester/SAPHanaSR-tests-angi-ScaleUp.7 index 2f608571..e18ccffd 100644 --- a/man-tester/SAPHanaSR-tests-angi-ScaleUp.7 +++ b/man-tester/SAPHanaSR-tests-angi-ScaleUp.7 @@ -1,6 +1,6 @@ .\" Version: 1.001 .\" -.TH SAPHanaSR-tests-angi-ScaleUp 7 "20 Nov 2023" "" "SAPHanaSR-angi" +.TH SAPHanaSR-tests-angi-ScaleUp 7 "20 Dec 2023" "" "SAPHanaSR-angi" .\" .SH NAME SAPHanaSR-tests-angi-ScaleUp \- Functional tests for SAPHanaSR Scale-Up. @@ -10,311 +10,88 @@ SAPHanaSR-tests-angi-ScaleUp \- Functional tests for SAPHanaSR Scale-Up. .PP Functional test are shipped for scale-up scenarios. This tests could be run out-of-the-box. The test cases are defined in dedicated files. -See manual page SAPHanaSR-tests-syntax(5) for syntax details. - -Entry point for all predefined tests is a clean and idle Linux cluster and a -clean HANA pair in sync. Same is true for the final state. -See manual page SAPHanaSR_maintenance_examples(7) for detecting the correct -status and watching changes near real-time. +See manual page SAPHanaSR-tests-syntax(5) for syntax details. Details like +performed steps or expected behaviour of cluster and HANA are explained in +SAPHanaSR-tests-description(7). .PP Predefined functional tests for scale-up overview: -.PP +.TP \fBblock_manual_takeover\fP -.RS 2 -Descr: blocked manual takeover, for susTkOver.py -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: See susTkOver.py(7). -.br -Expect: Both nodes stay online. -Both HANA stay online. -Failed manual takeover attempt is logged in syslog and HANA tracefile. -SR stays SOK. -No takeover. No fencing. -.RE -.PP +Blocked manual takeover, for susTkOver.py. +.TP \fBblock_sr\fP -.RS 2 -Descr: block HANA SR and check SFAIL attribute; unblock to recover -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: See susHanaSR.py.(7). -.br -Expect: Both nodes stay online. -Both HANA stay online. -SR SFAIL and finally SOK. -No takeover. No fencing. -.RE -.PP +Block HANA SR and check SFAIL attribute; unblock to recover. +.TP \fBblock_sr_and_freeze_prim_fs\fP -.RS 2 -Descr: block HANA SR and freeze HANA FS on primary master node -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: See ocf_suse_SAPHanaFilesystem(7), susHanaSR.py.(7). -.br -Expect: Both nodes stay online. -HANA primary is stopped and finally back online. -SR SFAIL and finally SOK. -No takeover. No fencing. -.RE -.PP +Block HANA SR and freeze HANA FS on primary master node. +.TP \fBflup\fP -.RS 2 -Descr: like nop but very short sleep only - only for checking the test engine -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: Wait and see. -.br -Expect: Cluster and HANA are up and running, all good. -.RE -.PP +Like nop but very short sleep, just checking the test engine. +.TP \fBfree_log_area\fP -.RS 2 -Descr: free HANA log area on primary site -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: Free up HANA transaction log space and log backups. -.br -Expect: Cluster and HANA are up and running, all good. -.RE -.PP +Free HANA log area on primary site. +.TP \fBfreeze_prim_fs\fP -.RS 2 -Descr: freeze HANA FS on primary master node -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: See ocf_suse_SAPHanaFilesystem(7). -.br -Expect: Primary node fenced and finally started as secondary. -HANA primary stopped and finally started as secondary. -HANA secondary becomes finally primary. -SR SFAIL and finally SOK. -One takeover. One fence. -.RE -.PP +Freeze HANA FS on primary master node. +.TP \fBkill_prim_indexserver\fP -.RS 2 -Descr: Kill primary indexserver, for susChkSrv.py -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: See susChkSrv.py(7). -.br -Expect: Primary node stays online. -HANA primary stopped and finally started as secondary. -HANA secondary becomes finally primary. -SR SFAIL and finally SOK. -One takeover. No fencing (for action_on_lost=kill). -.RE -.PP +Kill primary indexserver, for susChkSrv.py. +.TP \fBkill_prim_inst\fP -.RS 2 -Descr: Kill primary instance -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: HDB kill -.br -Expect: Primary node stays online. -HANA primary stopped and finally started as secondary. -HANA secondary becomes finally primary. -SR SFAIL and finally SOK. -One takeover. No fencing. -.RE -.PP +Kill primary instance. +.TP \fBkill_prim_node\fP -.RS 2 -Descr: Kill primary node -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: systemctl reboot --force -.br -Expect: Primary node fenced and finally started as secondary. -HANA primary stopped and finally started as secondary. -HANA secondary becomes finally primary. -SR SFAIL and finally SOK. -One takeover. One fencing. -.RE -.PP +Kill primary node. +.TP \fBkill_secn_indexserver\fP -.RS 2 -Descr: Kill secondary indexserver, for susChkSrv.py -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: See susChkSrv.py(7). -.br -Expect: HANA secondary stopped and finally online. -HANA primary stays online. -SR SFAIL and finally SOK. -No takeover. No fencing (for action_on_lost=kill). -.RE -.PP +Kill secondary indexserver, for susChkSrv.py. +.TP \fBkill_secn_inst\fP -.RS 2 -Descr: Kill secondary instance -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: HDB kill -.br -Expect: HANA secondary stopped and finally online. -HANA primary stays online. -SR SFAIL and finally SOK. -No takeover. No fencing. -.RE -.PP +Kill secondary instance. +.TP \fBkill_secn_node\fP -.RS 2 -Descr: Kill secondary node -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: systemctl reboot --force -.br -Expect: Secondary node fenced and finally online. -HANA primary stays online. -SR SFAIL and finally SOK. -No takeover. One fencing. -.RE -.PP +Kill secondary node. +.TP \fBmaintenance_cluster_turn_hana\fP -.RS 2 -Descr: maintenance procedure, manually turning HANA sites -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: See SAPHanaSR_maintenance_examples(7), https://www.suse.com/c/sap-hana-maintenance-suse-clusters/ . -.br -Expect: Both nodes stay online. -HANA primary stopped and finally started as secondary. -HANA secondary becomes finally primary by manual takeover. -SR SFAIL and finally SOK. -One takeover. No fencing. -.RE -.PP +Maintenance procedure, manually turning HANA sites. +.TP \fBmaintenance_with_standby_nodes\fP -.RS 2 -Descr: standby+online secondary then standby+online primary -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: See SAPHanaSR_maintenance_examples(7. -.br -Expect: Both nodes stay online. -HANA primary stopped and finally started as secondary. -HANA secondary becomes finally primary. -SR SFAIL and finally SOK. -One takeover. No fencing. -.RE -.PP +Maintenance procedure, standby+online secondary then standby+online primary. +.TP \fBnop\fP -.RS 2 -Descr: no operation - check, wait and check again (stability check) -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: Wait and see. -.br -Expect: Cluster and HANA are up and running, all good. -.RE -.PP +No operation - check, wait and check again (stability check). +.TP +\fBregister_prim_cold_hana\fP +Stop cluster, do manual takeover, leave former primary down and unregistered, start cluster (not yet implementeed). +.TP \fBrestart_cluster_hana_running\fP -.RS 2 -Descr: -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: -.br -Expect: Both nodes stay online. - -No takeover. No fencing. -.RE -.PP +Stop and restart cluster, keep HANA running. +.TP \fBrestart_cluster\fP -.RS 2 -Descr: -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: -.br -Expect: Both nodes stay online. - -No takeover. No fencing. -.RE -.PP +Stop and restart cluster and HANA. +.TP \fBrestart_cluster_turn_hana\fP -.RS 2 -Descr: -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: -.br -Expect: Both nodes stay online. - -One takeover. No fencing. -.RE +Stop cluster and HANA, manually start and takeover HANA, start cluster. .PP \fBsplit_brain_prio\fP -.RS 2 -Descr: Network split-brain with priority fencing -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: -.br -Expect: Secondary node fenced and finally online. -Primary node stays online. -HANA primary stays online. -SR SFAIL and finally SOK. -No takeover. One fencing. -.RE +Network split-brain with priority fencing. +.TP +\fBstandby_prim_node\fP +Set primary node standby and online again. +.TP +\fBstandby_secn_node\fP +Set secondary node standby and online again. .PP -\fBstandby_primary_node\fP -.RS 2 -Descr: Set primary node standby and online again -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: crm node standby ; crm node online -.br -Expect: Both nodes aty online. -Primary node standby and finally back online. -HANA primary stopped and finally started as secondary. -HANA secondary finally primary by takeover. -SR SFAIL and finally SOK. -One takeover. No fencing. -.RE +.\" +.SH EXAMPLES +.PP +* List tests for SAPHanaSR-angi scale-up scenarios .PP -\fBstandby_secondary_node\fP .RS 2 -Descr: Set secondary node standby and online again -.br -Prereq: Cluster and HANA are up and running, all good. -.br -Test: crm node standby ; crm node online -.br -Expect: Secondary node standby and finally online. -HANA primary stays online. -HANA secondary stopped and finally started. -SR SFAIL and finally SOK. No takeover. No fencing. +# ls /usr/share/SAPHanaSR-tester/json/angi-ScaleUp/ .RE .PP .\" -.SH EXAMPLES -.PP -.\" .SH FILES .\" .TP @@ -328,6 +105,8 @@ shell scripts for un-easy tasks on the cluster nodes. .SH REQUIREMENTS .\" See the REQUIREMENTS section in SAPHanaSR-tester(7) and SAPHanaSR-angi(7). +Of course, HANA database and Linux cluster have certain requirements. +Please refer to the product documentation. .PP .\" .SH BUGS @@ -338,7 +117,8 @@ Please report any other feedback and suggestions to feedback@suse.com. .\" .SH SEE ALSO \fBSAPHanaSR-tester\fP(7) , \fBSAPHanaSR-testCluster\fP(8) , -\fBSAPHanaSR-tests-angi-ScaleOut\fP(7) , \fBSAPHanaSR-tests-syntax\fP(5) , +\fBSAPHanaSR-tests-description\fP(7) , \fBSAPHanaSR-tests-syntax\fP(5) , +\fBSAPHanaSR-tests-angi-ScaleOut\fP(7) , \fBSAPHanaSR-angi\fP(7) , \fBSAPHanaSR-showAttr\fP(8) .PP .\" @@ -347,7 +127,7 @@ F.Herschel, L.Pinne. .PP .\" .SH COPYRIGHT -(c) 2023 SUSE Linux GmbH, Germany. +(c) 2023 SUSE LLC .br The package SAPHanaSR-tester comes with ABSOLUTELY NO WARRANTY. .br diff --git a/man-tester/SAPHanaSR-tests-basic-cluster.7 b/man-tester/SAPHanaSR-tests-basic-cluster.7 index 14c3f91a..1cd9f603 100644 --- a/man-tester/SAPHanaSR-tests-basic-cluster.7 +++ b/man-tester/SAPHanaSR-tests-basic-cluster.7 @@ -23,11 +23,23 @@ The checks outlined here are by no means complete. Neverteless, at least this checks should be performed before doing functional tests. .PP +\fB*\fR Checking access to update channels and pending patches. +.PP +The accessible update channels are shown. Also pending patches are listed. +Do this on all Linux cluster nodes. +See manual page zypper(8). +.PP +.RS 2 +# zypper lr +.br +# zypper lp +.RE +.PP \fB*\fR Checking systemd integration and OS settings for SAP HANA. .PP Basic OS features time and name resolution as well as HANA disk space are checked. Integration of SAP hostagent and SAP HANA database with systemd are -checked. OS settings for HANA are checked. +checked. OS settings for HANA are checked. Do this on all Linux cluster nodes. Cluster nodes are "node1" and "node2", HANA SID is "HA1", instance number is "85". Do this on all Linux cluster nodes. See manual page chronyc(8) and saptune(8). @@ -47,6 +59,7 @@ See manual page chronyc(8) and saptune(8). \fB*\fR Checking for errors in the system logs. .PP Known error patterns are looked up in the system log. +Do this on all Linux cluster nodes. See manual page cs_show_error_patterns(8). .PP .RS 2 @@ -125,7 +138,8 @@ The Linux sudo rules are checked for needed permissions. The HANA global.ini is checked for SUSE HA/DR provider hook scripts. The HANA nameserver tracefiles are checked whether the hook scripts have been loaded. SID is HA1, instance number is 85. Do this on all Linux cluster nodes. See -manual page susHanaSR.py(7), susTkOver.py(7) and susChkSrv.py(7). +manual page susHanaSR.py(7), susTkOver.py(7), susChkSrv.py(7) and +SAPHanaSR-hookHelper(8). .PP .RS 2 # sudo -U ha1adm -l | \\ @@ -181,6 +195,8 @@ path to HANA tracefiles .SH REQUIREMENTS .\" See the REQUIREMENTS section in SAPHanaSR-tester(7) and SAPHanaSR-angi(7). +Of course, the Linux cluster has certain requirements. +Please refer to the product documentation. .PP .\" .SH BUGS @@ -193,7 +209,7 @@ Please report any other feedback and suggestions to feedback@suse.com. \fBSAPHanaSR-tester\fP(7) , \fBSAPHanaSR-angi\fP(7) , \fBSAPHanaSR_basic_cluster\fP(7) , \fBSAPHanaSR-ScaleOut_basic_cluster\fP(7) , \fBsusHanaSR.py\fP(7) , \fBsusTkOver.py\fP(7) , \fBsusChkSrv.py\fP(7) , -\fBcrm\fP(8) , \fBcrm_verify\fP(8) , \fBstonith_admin\fP(8) , +\fBcrm\fP(8) , \fBcrm_verify\fP(8) , \fBcrm_mon\fP(8) , \fBstonith_admin\fP(8) , \fBcs_show_error_patterns\fP(8) , \fBcs_sum_base_config\fP(8) , \fBcs_show_sbd_devices\fP(8) , \fBsbd\fP(8) , \fBstonith_sbd\fP(8) , \fBsaptune\fP(8) , \fBchronyc\fP(8) , \fBsystemctl\fP(8) , \fBhosts\fP(5) , @@ -213,7 +229,7 @@ F.Herschel, L.Pinne. .PP .\" .SH COPYRIGHT -(c) 2023 SUSE Linux GmbH, Germany. +(c) 2023 SUSE LLC .br The package SAPHanaSR-tester comes with ABSOLUTELY NO WARRANTY. .br diff --git a/man-tester/SAPHanaSR-tests-description.7 b/man-tester/SAPHanaSR-tests-description.7 new file mode 100644 index 00000000..26e9848a --- /dev/null +++ b/man-tester/SAPHanaSR-tests-description.7 @@ -0,0 +1,705 @@ +.\" Version: 1.001 +.\" +.TH SAPHanaSR-tests-description 7 "04 Jan 2024" "" "SAPHanaSR-angi" +.\" +.SH NAME +SAPHanaSR-tests-description \- Functional tests for SAPHanaSR. +.PP +.\" +.SH DESCRIPTION +.PP +Functional test are shipped for different scenarios. This tests could be run +out-of-the-box. The test cases are defined in dedicated files. +See manual page SAPHanaSR-tests-syntax(5) for syntax details. Tests for +SAPHanaSR-angi scale-up scenarios are listed in SAPHanaSR-tests-angi-ScaleUp(7), +for SAPHanaSR-angi scale-out ERP scenarios in SAPHanaSR-tests-angi-ScaleOut(7). + +Entry point for all predefined tests is a clean and idle Linux cluster and a +clean HANA pair in sync. Same is true for the final state. +See manual page SAPHanaSR_maintenance_examples(7) for detecting the correct +status and watching changes near real-time. + +Each test can be executed by running the command SAPHanaSR-testCluster with +appropriate parameters. See manual page SAPHanaSR-testCluster(8). +.PP +Predefined functional tests: +.PP +\fBblock_manual_takeover\fP +.RS 2 +Descr: Blocked manual takeover, for susTkOver.py. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See susTkOver.py(7). +.br +Expect: Both nodes stay online. +Both HANA stay online. +Failed manual takeover attempt is logged in syslog and HANA tracefile. +SR stays SOK. +No takeover. No fencing. +.br +Comment: Admin mistake. +.RE +.PP +\fBblock_sr\fP +.RS 2 +Descr: Block HANA SR and check SFAIL attribute; unblock to recover. +.br +Topology: ScaleUp, ScaleOut (not yet implemented). +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See susHanaSR.py.(7). +.br +Expect: All nodes stay online. +Both HANA stay online. +SR SFAIL and finally SOK. +No takeover. No fencing. +.br +Comment: Infrastructure failure, main cluster case. +.RE +.PP +\fBblock_sr_and_freeze_prim_fs\fP +.RS 2 +Descr: Block HANA SR and freeze HANA FS on primary node. +.br +Topology: ScaleUp. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See ocf_suse_SAPHanaFilesystem(7), susHanaSR.py.(7). +.br +Expect: Both nodes stay online. +HANA primary is stopped and finally back online. +SR SFAIL and finally SOK. +No takeover. No fencing. +.br +Comment: Infrastructure failure, main cluster case. +.RE +.PP +\fBblock_sr_and_freeze_prim_master_nfs\fP +.RS 2 +Descr: Block HANA SR and freeze HANA NFS on primary master node +(not yet implemented). +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See ocf_suse_SAPHanaFilesystem(7), susHanaSR.py.(7). +.br +Expect: All nodes stay online. +HANA primary is stopped and finally back online. +SR SFAIL and finally SOK. +No takeover. No fencing. +.br +Comment: Infrastructure failure, main cluster case. +.RE +.PP +\fBblock_sr_and_freeze_prim_site_nfs\fP +.RS 2 +Descr: Block HANA SR and freeze HANA NFS on primary site +(not yet implemented). +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See ocf_suse_SAPHanaFilesystem(7), susHanaSR.py.(7). +.br +Expect: All nodes stay online. +HANA primary is stopped and finally back online. +SR SFAIL and finally SOK. +No takeover. No fencing. +.br +Comment: Infrastructure failure, main cluster case. +.RE +.PP +\fBflup\fP +.RS 2 +Descr: Like nop but very short sleep, just checking the test engine. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: Wait and see. +.br +Expect: Cluster and HANA are up and running, all good. +.br +Comment: Just housekeeping. +.RE +.PP +\fBfree_log_area\fP +.RS 2 +Descr: Free HANA log area on primary site. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: Free up HANA transaction log space and log backups. +.br +Expect: Cluster and HANA are up and running, all good. +.br +Comment: Just housekeeping. +.RE +.PP +\fBfreeze_prim_fs\fP +.RS 2 +Descr: Freeze HANA FS on primary node. +.br +Topology: ScaleUp. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See ocf_suse_SAPHanaFilesystem(7). +.br +Expect: Primary node fenced and finally started as secondary. +HANA primary stopped and finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. One fence. +.br +Comment: Infrastructure failure, main cluster case. +.RE +.PP +\fBfreeze_prim_master_nfs\fP +.RS 2 +Descr: Freeze HANA NFS on primary master node. +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See ocf_suse_SAPHanaFilesystem(7). +.br +Expect: Primary master node fenced, primary worker instance stopped. +HANA finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. One fence. +.br +Comment: Infrastructure failure, main cluster case. +.RE +.PP +\fBfreeze_prim_site_nfs\fP +.RS 2 +Descr: Freeze HANA NFS on primary site. +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See ocf_suse_SAPHanaFilesystem(7). +.br +Expect: Primary master an worker node fenced. +HANA finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. One fence. +.br +Comment: Infrastructure failure, main cluster case. +.RE +.PP +\fBfreeze_secn_site_nfs\fP +.RS 2 +Freeze HANA NFS on secondary site. +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See ocf_suse_SAPHanaFilesystem(7). +.br +Expect: Primary site keeps running. +HANA secondary site restarted. +SR SFAIL and finally SOK. +No takeover. No fence. +.br +Comment: Infrastructure failure, main cluster case. +.RE +.PP +\fBkill_prim_indexserver\fP +.RS 2 +Descr: Kill primary indexserver, for susChkSrv.py. +On scale-out, kill primary master indexserver. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See susChkSrv.py(7). +.br +Expect: Primary node stays online. +HANA primary (master) stopped and finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. No fencing (for action_on_lost=kill). +.br +Comment: Application failure, main cluster case. +.RE +.PP +\fBkill_prim_inst\fP +.RS 2 +Descr: Kill primary instance. +On scale-out, kill primary master instance. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: HDB kill +.br +Expect: Primary (master) node stays online. +HANA primary (master) stopped and finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. No fencing. +.br +Comment: Application failure, main cluster case. +.RE +.PP +\fBkill_prim_node\fP +.RS 2 +Descr: Kill primary node. +On scale-out, kill primary master node. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: systemctl reboot --force +.br +Expect: Primary (master) node fenced and finally started as secondary. +HANA primary stopped and finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. One fencing. +.br +Comment: Node failure, main cluster case. +.RE +.PP +\fBkill_prim_worker_indexserver\fP +.RS 2 +Descr: Kill primary worker indexserver, for susChkSrv.py. +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See susChkSrv.py(7). +.br +Expect: HANA primary stopped and finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. No fencing (for action_on_lost=kill). +.br +Comment: Application failure, main cluster case. +.RE +.PP +\fBkill_prim_worker_inst\fP +.RS 2 +Descr: Kill primary worker instance. +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: HDB kill +.br +Expect: HANA primary stopped and finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. No fencing. +.br +Comment: Application failure, main cluster case. +.RE +.PP +\fBkill_prim_worker_node\fP +.RS 2 +Descr: Kill primary worker node. +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: systemctl reboot --force +.br +Expect: Primary worker node fenced. +HANA primary stopped and finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. One fencing. +.br +Comment: Node failure, main cluster case. +.RE +.PP +\fBkill_secn_indexserver\fP +.RS 2 +Descr: Kill secondary indexserver, for susChkSrv.py. +On scale-out, kill secondary master indexserver. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See susChkSrv.py(7). +.br +Expect: HANA secondary stopped and finally online. +HANA primary stays online. +SR SFAIL and finally SOK. +No takeover. No fencing (for action_on_lost=kill). +.br +Comment: Application failure, main cluster case. +.RE +.PP +\fBkill_secn_inst\fP +.RS 2 +Descr: Kill secondary instance. +On scale-out, kill secondary master instance. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: HDB kill +.br +Expect: HANA secondary stopped and finally online. +HANA primary stays online. +SR SFAIL and finally SOK. +No takeover. No fencing. +.br +Comment: Application failure, main cluster case. +.RE +.PP +\fBkill_secn_node\fP +.RS 2 +Descr: Kill secondary node. +On scale-out, kill secondary master node. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: systemctl reboot --force +.br +Expect: Secondary (master) node fenced and finally online. +HANA primary stays online. +SR SFAIL and finally SOK. +No takeover. One fencing. +.br +Comment: Node failure, main cluster case. +.RE +.PP +\fBkill_secn_worker_inst\fP +.RS 2 +Descr: Kill secondary worker instance. +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: +.br +Expect: + +HANA primary stays online. +SR SFAIL and finally SOK. +No takeover. No fencing. +.br +Comment: Application failure, main cluster case. +.RE +.PP +\fBkill_secn_worker_node\fP +.RS 2 +Descr: Kill secondary worker node. +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: systemctl reboot --force +.br +Expect: Secondary worker node fenced and finally online. +HANA primary stays online. +SR SFAIL and finally SOK. +No takeover. One fencing. +.br +Comment: Node failure, main cluster case. +.RE +.PP +\fBmaintenance_cluster_turn_hana\fP +.RS 2 +Descr: Maintenance procedure, manually turning HANA sites. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See SAPHanaSR_maintenance_examples(7), https://www.suse.com/c/sap-hana-maintenance-suse-clusters/ . +.br +Expect: All nodes stay online. +HANA primary stopped and finally started as secondary. +HANA secondary becomes finally primary by manual takeover. +SR SFAIL and finally SOK. +One takeover. No takeover by cluster. No fencing. +.br +Comment: Main admin procedure. +.RE +.PP +\fBmaintenance_with_standby_nodes\fP +.RS 2 +Descr: standby+online secondary then standby+online primary +.br +Topology: ScaleUp. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: See SAPHanaSR_maintenance_examples(7). +.br +Expect: All nodes stay online. +HANA primary stopped and finally started as secondary. +HANA secondary becomes finally primary. +SR SFAIL and finally SOK. +One takeover. No fencing. +.br +Comment: Sub-optimal admin procedure. +.RE +.PP +\fBnop\fP +.RS 2 +Descr: No operation - check, wait and check again (stability check). +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: Wait and see. +.br +Expect: Cluster and HANA are up and running, all good. +.br +Comment: Main cluster case. +.RE +.PP +\fBregister_prim_cold_hana\fP +.RS 2 +Descr: Stop cluster, do manual takeover, leave former primary down and unregistered, start cluster. +.br +Topology: ScaleUp, ScaleOut (not yet implemented). +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: +.br +Expect: All nodes stay online. +HANA primary stopped and finally started as secondary. +HANA secondary stopped and finally started as primary. +SR SFAIL and finally SOK. +One takeover. No takeover by cluster. No fencing. +.br +Comment: Admin mistake. +.RE +.PP +\fBrestart_cluster_hana_running\fP +.RS 2 +Descr: Stop and restart cluster, keep HANA running. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: crm maintenance on; +crm cluster run "crm cluster stop"; +crm cluster run "crm cluster start"; +crm resource refresh ; +crm resource refresh ; +crm resource maintenance off; +.br +Expect: All nodes stay online. +Cluster stopped and restarted. +Both HANA keep running. +SR stays SOK. +No takeover. No fencing. +.br +Comment: Main admin procedure. +.RE +.PP +\fBrestart_cluster\fP +.RS 2 +Descr: Stop and restart cluster and HANA. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: crm cluster run "crm cluster stop"; +sapcontrol ... StartSystem; +sapcontrol ... StartSystem; +crm cluster run "crm cluster start"; +.br +Expect: All nodes stay online. +Cluster stopped and restarted. +Both HANA stopped and manually restarted. +SR SFAIL and finally SOK. +No takeover. No fencing. +.br +Comment: Sub-optimal admin procedure. +.RE +.PP +\fBrestart_cluster_turn_hana\fP +.RS 2 +Descr: Stop cluster and HANA, manually start and takeover HANA, start cluster. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: Stop cluster and HANA, manually start HANA and do takeover, restart cluster. +No resource maintenance, no resource refresh. +.br +Expect: All nodes stay online. +Both HANA stopped. +HANA primary finally started as secondary. +HANA secondary becomes finally primary by manual takeover. +SR SFAIL and finally SOK. +One takeover. No takeover by cluster. No fencing. +.br +Comment: Sub-optimal admin procedure, challenge for susHanaSR.py. +.RE +.PP +\fBsplit_brain_prio\fP +.RS 2 +Descr: Network split-brain with priority fencing. +.br +Topology: ScaleUp. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: iptables -I INPUT -s -j DROP +.br +Expect: Secondary node fenced and finally online. +Primary node stays online. +HANA primary stays online. +SR SFAIL and finally SOK. +No takeover. One fencing. +.br +Comment: Infrastructure failure, main cluster case. +.RE +.PP +\fBstandby_prim_node\fP +.RS 2 +Descr: Set primary node standby and online again. +On scale-out, standby primary master node and online again. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: crm node standby ; crm node online +.br +Expect: All nodes stay online. +Primary (master) node standby and finally back online. +HANA primary stopped and finally started as secondary. +HANA secondary finally primary by takeover. +SR SFAIL and finally SOK. +One takeover. No fencing. +.br +Comment: Admin mistake on scale-out, sub-optimal procedure on scale-up. +.RE +.PP +\fBstandby_secn_node\fP +.RS 2 +Descr: Set secondary node standby and online again. +On scale-out, standby secondary master node and online again. +.br +Topology: ScaleUp, ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: crm node standby ; crm node online +.br +Expect: Secondary (master) node standby and finally online. +HANA primary stays online. +HANA secondary stopped and finally started. +SR SFAIL and finally SOK. No takeover. No fencing. +.br +Comment: Admin mistake on scale-out, sub-optimal procedure on scale-up. +.RE +.PP +\fBstandby_secn_worker_node\fP +.RS 2 +Descr: Set secondary worker node standby and online again. +.br +Topology: ScaleOut. +.br +Prereq: Cluster and HANA are up and running, all good. +.br +Test: crm node standby ; crm node online +.br +Expect: Secondary worker node standby and finally online. +HANA primary stays online. +HANA secondary stays online. HANA worker clone_state goes to UNDEFINED and +finally to DEMOTED. +SR stays SOK. No takeover. No fencing. +.br +Comment: Admin mistake. +.RE +.PP +.\" +.SH EXAMPLES +.PP +* List all shipped tests +.PP +.RS 2 +# find /usr/share/SAPHanaSR-tester/json/ -name "*.json" -exec basename {} \\; | sort -u +.RE +.PP +.\" +.SH FILES +.\" +.TP +/usr/share/SAPHanaSR-tester/json/angi-ScaleUp/ +functional tests for SAPHanaSR-angi scale-up scenarios. +.TP +/usr/share/SAPHanaSR-tester/json/angi-ScaleOut/ +functional tests for SAPHanaSR-angi scale-out ERP scenarios. +.TP +/usr/bin/sct_test_* +shell scripts for un-easy tasks on the cluster nodes. +.PP +.\" +.SH REQUIREMENTS +.\" +See the REQUIREMENTS section in SAPHanaSR-tester(7) and SAPHanaSR-angi(7). +Of course, HANA database and Linux cluster also have certain requirements. +Please refer to the product documentation. +.PP +.\" +.SH BUGS +In case of any problem, please use your favourite SAP support process to open +a request for the component BC-OP-LNX-SUSE. +Please report any other feedback and suggestions to feedback@suse.com. +.PP +.\" +.SH SEE ALSO +\fBSAPHanaSR-tester\fP(7) , \fBSAPHanaSR-testCluster\fP(8) , +\fBSAPHanaSR-tests-syntax\fP(5) , \fBSAPHanaSR-tests-angi-ScaleUp\fP(7) , +\fBSAPHanaSR-tests-angi-ScaleOut\fP(7) , \fBSAPHanaSR-angi\fP(7) , +\fBSAPHanaSR-showAttr\fP(8) +.PP +.\" +.SH AUTHORS +F.Herschel, L.Pinne. +.PP +.\" +.SH COPYRIGHT +(c) 2023-2024 SUSE LLC +.br +The package SAPHanaSR-tester comes with ABSOLUTELY NO WARRANTY. +.br +For details see the GNU General Public License at +http://www.gnu.org/licenses/gpl.html +.\" diff --git a/man-tester/SAPHanaSR-tests-syntax.5 b/man-tester/SAPHanaSR-tests-syntax.5 index fbbd6571..79272ef2 100644 --- a/man-tester/SAPHanaSR-tests-syntax.5 +++ b/man-tester/SAPHanaSR-tests-syntax.5 @@ -29,11 +29,76 @@ SAPHanaSR-tests-angi-ScaleOut(7) for predefined test cases. .\" Test cases are defined in JSON files. Each test case is described in one file. It also can reference to defaults.json for common status definitions. + +TODO on first level +[ test | name | start | comment | steps ] +.\" TODO comment or remark? + +.TP +\fBtest\fP +test identifier, unique +.TP +\fBname\fP +test description +.TP +\fBstart\fP +name of step to start the test with +.TP +\fBcomment\fP +comment, just a string +.TP +\fBsteps\fP +test steps, see below .PP -\fB*\fP Test step status +\fBsteps\fP + +TODO +[ step | name | next | loop | wait | post | comment | ] + +.TP +\fBstep\fP +step identifier, unique +.TP +\fBname\fP +step description +.TP +\fBnext\fP +name of step to do next +.TP +\fBloop\fP +maximum number of loops to wait and check for status +.TP +\fBwait\fP +how many seconds to wait for each loop TODO +.TP +\fBcomment\fP +comment, just a string +.TP +\fBstatus\fP +status definition, see below + +.PP +\fB*\fP Test step status +.PP +Each step contains an expectation for SAPHanaSR attributes, which need to match. +The attributes are arranged along the CIB and SAPHanaSR-showAttr sections. +The + +[ pHost | sHost | pWorker | sWorker | pSite | sSite | pGlobal | sGlobal ] + +.TP +pHost, sHost, pWorker, sWorker +node attributes +.TP +pSite, sSite +site attributes +.TP +pGlobal, sGlobal +global attributes, not yet implmented +.PP +See manual page SAPHanaSR-showAttr(8) for details on that attributes. .PP -Each step contains an expectation for SAPHanaSR attributes, which need to -match. The step status are: +Some common status are predefind as defaults: .TP globalUp both sites are up and in sync @@ -42,19 +107,29 @@ globalDown not yet implemented .TP pHostUp -host p (expected to be primary before the test starts) is up +host p (expected to be primary before the test starts) is up (scale-out: expected primary master node) +.TP +pWorkerUp +worker node p is up (scale-out) .TP pSiteUp -site p is up +site p is up .TP sSiteUp site s (expected to be secondary before the test starts) is up .TP sHostUp -host s is up +host s is up (scale-out: master node) +.TP +sWorkerUp +worker node s is up (scale-out) .TP pHostDown -host p is down +host p is down (scale-out: master node) +.TP +pWorkerDown +worker node p is down (scale-out) +not yet implemented .TP pSiteDown site p is down @@ -63,7 +138,11 @@ sSiteDown site s is down .TP sHostDown -host s is down +host s is down (scale-out: master node) +.TP +sWorkerDown +worker node s is down (scale-out) +not yet implemented .PP Note: Prefixes "s" and "p" are statically indicating geographical sites, as seen at the beginning of a test. If a takeover happens during that test, the @@ -119,15 +198,87 @@ not like, regular expression is TODO, the string "None" .PP -\fB*\fP Test step actions +\fB*\fP Test step action "post" .PP TODO +.PP +action on cluster after step, one of: +.TP +bmt +block manual takeover +.TP +cleanup +cleanup Linux cluster resource failure +.TP +kill_prim_worker_node +kill pirmary worker node (scale-out) +.TP +kill_prim_node +kill pirmary node (scale-out: primary master node) +.TP +kill_secn_worker_node +kill secondary worker node (scale-out) +.TP +kill_secn_node +kill secondary node (scale-out: secondary master node) +.TP +kill_prim_inst +kill pirmary instance (scale-out: primary master instance) +.TP +kill_prim_worker_inst +kill worker pirmary instance (scale-out) +.TP +kill_secn_inst +kill secondary instance (scale-out: secondary master instance) +.TP +kill_secn_worker_inst +kill worker secondary instance (scale-out) +.TP +kill_prim_indexserver +kill primary indexserver (scale-out: primary master indexserver) .TP -post -action after step, [ bmt | cleanup | spn | ssn | osn | script