Version 1.9.0 (#391)

## [1.9.0]
### Added
- Added support for Python 3.11
- Update docker docs for new rabbitmq and redis server versions
- Added lgtm.com Badge for README.md
- More fixes for lgtm checks.
- Added merlin server command as a container option for broker and results_backend servers.
- Added new documentation for merlin server in docs and tutorial
- Added the flux_exec batch argument to allow for flux exec arguments,
  e.g. flux_exec: flux exec -r "0-1" to run celery workers only on
  ranks 0 and 1 of a multi-rank allocation
- Additional argument in test definitions to allow for a post "cleanup" command
- Capability for non-user block in yaml
- .readthedocs.yaml and requirements.txt files for docs
- Small modifications to the Tutorial, Getting Started, Command Line, and Contributing pages in the docs
- Compatibility with the newest version of Maestro (v. 1.1.9dev1)
- JSON schema validation for Merlin spec files
- New tests related to JSON schema validation
- Instructions in the "Contributing" page of the docs on how to add new blocks/fields to the spec file
- Brief explanation of the $(LAUNCHER) variable in the "Variables" page of the docs

### Changed
- Removed support for Python 3.6
- Rename lgtm.yml to .lgtm.yml
- New shortcuts in specification file (sample_vector, sample_names, spec_original_template, spec_executed_run, spec_archived_copy)
- Update requirements to require redis 4.3.4 for acl user channel support
- Added ssl to the broker and results backend server checks when "merlin info" is called
- Removed theme_override.css from docs/_static/ since it is no longer needed with the updated version of sphinx
- Updated docs/Makefile to include a pip install for requirements and a clean command
- Update to the Tutorial and Contributing pages in the docs
- Changed what is stored in a Merlin DAG
  - We no longer store the entire Maestro ExecutionGraph object
  - We now only store the adjacency table and values obtained from the ExecutionGraph object
- Modified spec verification
- Update to require maestrowf 1.9.1dev1 or later

### Fixed
- Fixed return values from scripts with main() to fix testing errors. 
- CI test for CHANGELOG modifcations
- Typo "cert_req" to "cert_reqs" in the merlin config docs
- Removed emoji from issue templates that were breaking doc builds
- Including .temp template files in MANIFEST
- Styling in the footer for docs
- Horizontal scroll overlap in the variables page of the docs
- Reordered small part of Workflow Specification page in the docs in order to put "samples" back in the merlin block

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Joseph M. Koning <[email protected]>
Co-authored-by: Joe Koning <[email protected]>
Co-authored-by: Ryan Lee <[email protected]>
Co-authored-by: Brian Gunnarson <[email protected]>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,13 +1,13 @@
 ---
-name: "\U0001F41B Bug report"
+name: "Bug report"
 about: Create a report to help us improve
 title: "[BUG] "
 labels: bug
 assignees: ''
 
 ---
 
-## 🐛 Bug Report
+## Bug Report
 
 **Describe the bug**
 A clear and concise description of what the bug is.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,5 +1,5 @@
 ---
-name: "\U0001F680 Feature request"
+name: "Feature request"
 about: Suggest an idea for Merlin
 title: "[FEAT] "
 labels: enhancement
@@ -9,7 +9,7 @@ assignees: ''
 
 <!-- NOTE: If your feature is related to a bug, please file a Bug report instead! -->
 
-## 🚀 Feature Request
+## Feature Request
 
 **What problem is this feature looking to solve?**
 A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

.github/ISSUE_TEMPLATE/question.md

@@ -1,9 +1,9 @@
 ---
-name: 🤓 General question
+name: General question
 labels: 'question'
 title: '[Q/A] '
 about: Ask, discuss, debate with the Merlin team
 ---
 
-## 🤓 Question
+## Question
 <!-- (Have you searched the issues page before asking?) -->

.github/workflows/push-pr_workflow.yml

@@ -14,7 +14,7 @@ jobs:
     - name: Check that CHANGELOG has been updated
       run: |
         # If this step fails, this means you haven't updated the CHANGELOG.md file with notes on your contribution.
-        git diff --name-only $(git merge-base origin/main HEAD) | grep '^CHANGELOG.md$' && echo "Thanks for helping keep our CHANGELOG up-to-date!"
+        git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.sha }} | grep '^CHANGELOG.md$' && echo "Thanks for helping keep our CHANGELOG up-to-date!"
 
   Lint:
     runs-on: ubuntu-latest
@@ -67,10 +67,15 @@ jobs:
 
   Local-test-suite:
     runs-on: ubuntu-latest
+    env: 
+      GO_VERSION: 1.18.1
+      SINGULARITY_VERSION: 3.9.9
+      OS: linux
+      ARCH: amd64
 
     strategy:
       matrix:
-        python-version: ['3.6', '3.7', '3.8', '3.9', '3.10']
+        python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
 
     steps:
     - uses: actions/checkout@v2
@@ -90,6 +95,27 @@ jobs:
         python3 -m pip install --upgrade pip
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
         pip3 install -r requirements/dev.txt
+    
+    - name: Install singularity
+      run: |
+        sudo apt-get update && sudo apt-get install -y \
+          build-essential \
+          libssl-dev \
+          uuid-dev \
+          libgpgme11-dev \
+          squashfs-tools \
+          libseccomp-dev \
+          pkg-config
+        wget https://go.dev/dl/go$GO_VERSION.$OS-$ARCH.tar.gz
+        sudo tar -C /usr/local -xzf go$GO_VERSION.$OS-$ARCH.tar.gz
+        rm go$GO_VERSION.$OS-$ARCH.tar.gz
+        export PATH=$PATH:/usr/local/go/bin
+        wget https://github.com/sylabs/singularity/releases/download/v$SINGULARITY_VERSION/singularity-ce-$SINGULARITY_VERSION.tar.gz
+        tar -xzf singularity-ce-$SINGULARITY_VERSION.tar.gz
+        cd singularity-ce-$SINGULARITY_VERSION
+        ./mconfig && \
+          make -C ./builddir && \
+          sudo make -C ./builddir install
 
     - name: Install merlin to run unit tests
       run: |
@@ -132,7 +158,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: ['3.6', '3.7', '3.8', '3.9', '3.10']
+        python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
 
     steps:
     - uses: actions/checkout@v2

.readthedocs.yaml

@@ -0,0 +1,13 @@
+version: 2
+
+build:
+  os: "ubuntu-20.04"
+  tools:
+    python: "3.8"
+
+sphinx:
+  configuration: docs/source/conf.py
+
+python:
+  install:
+    - requirements: docs/requirements.txt

CHANGELOG.md

@@ -4,13 +4,51 @@ All notable changes to Merlin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [unreleased]
+## [1.9.0]
 ### Added
+- Added support for Python 3.11
 - Update docker docs for new rabbitmq and redis server versions
 - Added lgtm.com Badge for README.md
 - More fixes for lgtm checks.
+- Added merlin server command as a container option for broker and results_backend servers.
+- Added new documentation for merlin server in docs and tutorial
+- Added the flux_exec batch argument to allow for flux exec arguments,
+  e.g. flux_exec: flux exec -r "0-1" to run celery workers only on
+  ranks 0 and 1 of a multi-rank allocation
+- Additional argument in test definitions to allow for a post "cleanup" command
+- Capability for non-user block in yaml
+- .readthedocs.yaml and requirements.txt files for docs
+- Small modifications to the Tutorial, Getting Started, Command Line, and Contributing pages in the docs
+- Compatibility with the newest version of Maestro (v. 1.1.9dev1)
+- JSON schema validation for Merlin spec files
+- New tests related to JSON schema validation
+- Instructions in the "Contributing" page of the docs on how to add new blocks/fields to the spec file
+- Brief explanation of the $(LAUNCHER) variable in the "Variables" page of the docs
+
 ### Changed
+- Removed support for Python 3.6
 - Rename lgtm.yml to .lgtm.yml
+- New shortcuts in specification file (sample_vector, sample_names, spec_original_template, spec_executed_run, spec_archived_copy)
+- Update requirements to require redis 4.3.4 for acl user channel support
+- Added ssl to the broker and results backend server checks when "merlin info" is called
+- Removed theme_override.css from docs/_static/ since it is no longer needed with the updated version of sphinx
+- Updated docs/Makefile to include a pip install for requirements and a clean command
+- Update to the Tutorial and Contributing pages in the docs
+- Changed what is stored in a Merlin DAG
+  - We no longer store the entire Maestro ExecutionGraph object
+  - We now only store the adjacency table and values obtained from the ExecutionGraph object
+- Modified spec verification
+- Update to require maestrowf 1.9.1dev1 or later
+
+### Fixed
+- Fixed return values from scripts with main() to fix testing errors. 
+- CI test for CHANGELOG modifcations
+- Typo "cert_req" to "cert_reqs" in the merlin config docs
+- Removed emoji from issue templates that were breaking doc builds
+- Including .temp template files in MANIFEST
+- Styling in the footer for docs
+- Horizontal scroll overlap in the variables page of the docs
+- Reordered small part of Workflow Specification page in the docs in order to put "samples" back in the merlin block
 
 ## [1.8.5]
 ### Added

CONTRIBUTORS

@@ -5,3 +5,4 @@ Benjamin Bay <[email protected]>
 Joe Koning <[email protected]>
 Jeremy White <[email protected]>
 Aidan Keogh
+Ryan Lee <[email protected]>

MANIFEST.in

@@ -1,4 +1,5 @@
 recursive-include merlin/data *.yaml *.py
-recursive-include merlin/examples *.py *.yaml *.c *.json *.sbatch *.bsub *.txt
+recursive-include merlin/server *.yaml *.py
+recursive-include merlin/examples *.py *.yaml *.c *.json *.sbatch *.bsub *.txt *.temp
 include requirements.txt
 include requirements/*

Makefile

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.8.4.
+# This file is part of Merlin, Version: 1.9.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -164,12 +164,12 @@ checks: check-style check-camel-case
 # automatically make python files pep 8-compliant
 fix-style:
 	. $(VENV)/bin/activate; \
-	isort --line-length $(MAX_LINE_LENGTH) $(MRLN); \
-	isort --line-length $(MAX_LINE_LENGTH) $(TEST); \
-	isort --line-length $(MAX_LINE_LENGTH) *.py; \
-	black --target-version py36 -l $(MAX_LINE_LENGTH) $(MRLN); \
-	black --target-version py36 -l $(MAX_LINE_LENGTH) $(TEST); \
-	black --target-version py36 -l $(MAX_LINE_LENGTH) *.py; \
+	$(PYTHON) -m isort --line-length $(MAX_LINE_LENGTH) $(MRLN); \
+	$(PYTHON) -m isort --line-length $(MAX_LINE_LENGTH) $(TEST); \
+	$(PYTHON) -m isort --line-length $(MAX_LINE_LENGTH) *.py; \
+	$(PYTHON) -m black --target-version py36 -l $(MAX_LINE_LENGTH) $(MRLN); \
+	$(PYTHON) -m black --target-version py36 -l $(MAX_LINE_LENGTH) $(TEST); \
+	$(PYTHON) -m black --target-version py36 -l $(MAX_LINE_LENGTH) *.py; \
 
 
 # Increment the Merlin version. USE ONLY ON DEVELOP BEFORE MERGING TO MASTER.

docs/Makefile

@@ -23,6 +23,9 @@ view: code-docs html
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
+	pip install -r requirements.txt
 	echo $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
+clean:
+	rm -rf build/

docs/requirements.txt

@@ -0,0 +1,56 @@
+#
+# This file is autogenerated by pip-compile with python 3.8
+# To update, run:
+#
+#    pip-compile requirements.in
+#
+alabaster==0.7.12
+    # via sphinx
+babel==2.10.3
+    # via sphinx
+certifi==2022.12.7
+    # via requests
+charset-normalizer==2.1.1
+    # via requests
+docutils==0.17.1
+    # via sphinx
+idna==3.4
+    # via requests
+imagesize==1.4.1
+    # via sphinx
+importlib-metadata==5.0.0
+    # via sphinx
+jinja2==3.0.3
+    # via sphinx
+markupsafe==2.1.1
+    # via jinja2
+packaging==21.3
+    # via sphinx
+pygments==2.13.0
+    # via sphinx
+pyparsing==3.0.9
+    # via packaging
+pytz==2022.5
+    # via babel
+requests==2.28.1
+    # via sphinx
+snowballstemmer==2.2.0
+    # via sphinx
+sphinx==5.3.0
+    # via -r requirements.in
+sphinxcontrib-applehelp==1.0.2
+    # via sphinx
+sphinxcontrib-devhelp==1.0.2
+    # via sphinx
+sphinxcontrib-htmlhelp==2.0.0
+    # via sphinx
+sphinxcontrib-jsmath==1.0.1
+    # via sphinx
+sphinxcontrib-qthelp==1.0.3
+    # via sphinx
+sphinxcontrib-serializinghtml==1.1.5
+    # via sphinx
+urllib3==1.26.12
+    # via requests
+zipp==3.10.0
+    # via importlib-metadata

docs/source/_static/custom.css

@@ -18,3 +18,22 @@ div.highlight .copybtn:hover {
 div.highlight {
     position: relative;
 }
+div.sphinxsidebar {
+    max-height: 100%;
+    overflow-y: auto;
+}
+td {
+	max-width: 300px;
+}
+@media screen and (min-width: 875px) {
+	.sphinxsidebar {
+		background-color: #fff;
+		margin-left: 0;
+		z-index: 1;
+		height: 100vh;
+		top: 0px;
+	}
+}
+.underline {
+    text-decoration: underline;
+}

docs/source/conf.py

@@ -47,7 +47,7 @@
 #    'sphinx.ext.autodoc',
 #    'sphinx.ext.intersphinx',
 # ]
-extensions = []
+extensions = ['sphinx.ext.autodoc']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -101,11 +101,7 @@
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
 
-html_context = {
-    'css_files': [
-        '_static/theme_overrides.css',  # override wide tables in RTD theme
-    ],
-}
+html_css_files = ['custom.css']
 
 # Custom sidebar templates, must be a dictionary that maps document names
 # to template names.
@@ -149,7 +145,7 @@
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
     (master_doc, 'Merlin.tex', u'Merlin Documentation',
-     u'MLSI', 'manual'),
+     u'The Merlin Development Team', 'manual'),
 ]
 
 
@@ -188,8 +184,6 @@ def setup(app):
     try:
         app.add_javascript("custom.js")
         app.add_javascript("https://cdn.jsdelivr.net/npm/clipboard@1/dist/clipboard.min.js")
-        app.add_stylesheet('custom.css')
     except AttributeError:
-        app.add_css_file('custom.css')
         app.add_js_file("custom.js")
         app.add_js_file("https://cdn.jsdelivr.net/npm/clipboard@1/dist/clipboard.min.js")