Develop (#83)

* pruned material from the README * trimmed readme again * added license section * remove cryptography easy_install in Makefile * MANIFEST.in edits * made over Makefile. Removed LC-specific material, moved all dependencies to requirements files. * This enables pip install 'merlinwf[dev]' . Main requirements.txt is just the release file. * Update CHANGELOG * Update travis test to pull in dev reqs * added pip install to getting started * fixed small omissions * added workflows variable * manifest updates * added to CHANGELOG * updated the CHANGELOG * added pip install to README * updated quick start * updated quick start * updated quick start * updated quick start * readme updates * removed redundant section * updated CHANGELOG * attempting to fix pypi image * removed broken (on PYPI) image * updated CHANGELOG * added title * made Sphinx docs version pull from merlin/__init__.py * Update MANIFEST.in * Remove app config from docs (#46) * removed app_config from docs * reverted app_amqp.yaml deletion, changed some paths * fixed image * install-tasks -> install-merlin * Update Makefile * Update getting_started.rst * Update conf.py * removed python2-only word 'unicode' * added to CHANGELOG * Add merlin status feature to query number of tasks and workers on queue server * Update getting_started.rst * Run make fix-style * added arxiv link to faq and README * restored .gitignore file * Break up spec queue list and queue list string * Add dump to csv option for status; useful for later parsing / monitoring. Prints results to screen instead of log. * Only write the header if opening a new file * Run black * Update CHANGELOG * restoring merlin-example changes * updated ci tests * fixed style * updated CHANGELOG * moved scripts into feature_demo * added merlin-example cli test * updated .travis.yml * corrected cli test ids * updated docs * fixed typo in docs * working on merlin-example -> merlin example * fixed style * cli command works * fixed style * updates to merlin example * cli tool is now fully-functional. * added a new cli test * updated travis test ids * updated docs * updated for uocoming merlin example feature * addressed reviwer comment * updated example workflow descriptions * fixed docs typo * fixed various typos * added to CHANGELOG * addressed reviewer comments * merlin-templates warning -> error * var name change, generalized 1-file rule for examples. * added to docs * Update README.md * incremented version Co-authored-by: Luc Peterson <[email protected]>
LLNL · Jan 8, 2020 · d547b84 · d547b84
1 parent 0b47ebd
commit d547b84
Show file tree

Hide file tree

Showing 94 changed files with 772 additions and 751 deletions.
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,68 @@
+# Merlin .gitignore
+
+# Python bytecode and serialized files
+__pycache__/
+*.py[cod]
+*.pkl
+*.db
+*.npy
+*.npz
+*.hdf5
+dist/
+build/
+
+# Egg packaging
+*.egg
+*.egg-info/
+
+# virtual environments
+toss3venv/
+venv/
+chaos5-venv/
+blueos/
+venv_*
+
+# Merlin-generated output
+*_ARCHIVE/
+*_ARCHIVE_DIR/
+ARCHIVE
+ARCHIVE_DIR
+*_OUTPUT/
+*_OUTPUT_D/
+*_ensemble_*/
+
+# Scheduler logs
+flux.out
+slurm*.out
+docs/build/
+
+# Tox files
+.tox/*
+
+# Jupyter
+jupyter/.ipynb_checkpoints
+jupyter/testDistributedSamples.py
+
+# Images
+*.png
+*.pdf
+
+# aspell
+*.bak
+
+# Misc file extension types
+*.swl
+*.swm
+*.swn
+*.swo
+*.swp
+*.db
+*.npy
+*.log
+
+# IDEs
+*.idea
+
+# Misc directories
+dist/
+build/
diff --git a/.travis.yml b/.travis.yml
@@ -5,16 +5,16 @@ python:
   - "3.8"
 install: 
   - "pip install --upgrade pip"
-  - "pip install cryptography"
   - "pip install -r requirements.txt"
-  - "pip install -r workflows/feature_demo/requirements.txt"
+  - "pip install -r merlin/examples/workflows/feature_demo/requirements.txt"
   - "pip install -e ."
   - "pip install -r requirements/mysql.txt"
+  - "pip install -r requirements/dev.txt"
   - "pip install --upgrade sphinx"
 script: 
   - "merlin config"
   - "python -m pytest tests/"
-  - "python tests/integration/run_tests.py --verbose --ids 1 2 3 4 5 6 7 8 9 12 13 18 19 20"
+  - "python tests/integration/run_tests.py --verbose --ids 1 2 3 4 5 6 7 8 9 12 13 14 15 20 21 22"
 deploy:
   provider: pypi
   user: "__token__"

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,33 @@ All notable changes to Merlin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.0] - 2020-01-07
+
+### Added
+- Development dependencies install via pip: `pip install "merlinwf[dev]"`.
+- `merlin status <yaml spec>` that returns queues, number of connected
+  workers and number of unused tasks in each of those queues.
+- `merlin example` cli command, which allows users to start running the
+  examples immedately (even after pip-installing).
+
+### Fixed
+- `MANIFEST.in` fixes as required by Spack.
+- `requirements.txt` just has release components, not dev deps.
+- A bug related to the deprecated word 'unicode' in `openfilelist.py`.
+- Broken Merlin logo image on PyPI summary page.
+- Documentation typos.
+
+### Changed
+- Made `README.md` more concise and user-friendly.
+
+### Removed
+- Dependencies outside the requirements directory.
+- LLNL-specific material in the Makefile.
+
+### Deprecated
+- `merlin-templates` cli command, in favor of `merlin example`.
+
+
 ## [1.0.5] - 2019-12-05
 
 ### Fixed

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -1,5 +1,3 @@
-prune docs
-prune tests
 include workflows/*
 include merlin/data/*
 include requirements.txt

diff --git a/Makefile b/Makefile
@@ -31,22 +31,21 @@
 PYTHON?=python3
 PYV=$(shell $(PYTHON) -c "import sys;t='{v[0]}_{v[1]}'.format(v=list(sys.version_info[:2]));sys.stdout.write(t)")
 PYVD=$(shell $(PYTHON) -c "import sys;t='{v[0]}.{v[1]}'.format(v=list(sys.version_info[:2]));sys.stdout.write(t)")
-VENV?=venv_merlin_$(SYS_TYPE)_py$(PYV)
-CERT?=/etc/pki/tls/cert.pem
+VENV?=venv_merlin_py$(PYV)
 PIP?=$(VENV)/bin/pip
-MRLN?=merlin/
-TEST?=tests/
+MRLN=merlin/
+TEST=tests/
+WKFW=merlin/examples/workflows/
 MAX_COMPLEXITY?=5
-VENVMOD?=venv
 
 PENV=merlin$(PYV)
 
 .PHONY : all
-.PHONY : install
+.PHONY : install-dev
 .PHONY : virtualenv
+.PHONY : install-workflow-deps
 .PHONY : install-pip-mysql
-.PHONY : install-tasks
-.PHONY : install-scipy
+.PHONY : install-merlin
 .PHONY : update
 .PHONY : pull
 .PHONY : clean-output
@@ -59,40 +58,34 @@ PENV=merlin$(PYV)
 .PHONY : check-style
 .PHONY : check-camel-case
 .PHONY : checks
-.PHONY : start-workers
 
 
-all: install install-tasks install-pip-mysql install-sphinx
+all: install-dev install-merlin install-workflow-deps install-pip-mysql
 
 
 # install requirements
-install: virtualenv
-	$(VENV)/bin/easy_install cryptography
-	$(PIP) install --cert $(CERT) -r requirements.txt
+install-dev: virtualenv
+	$(PIP) install -r requirements/dev.txt
 
 
 # this only works outside the venv
 virtualenv:
-	$(PYTHON) -m $(VENVMOD) $(VENV) --prompt $(PENV) --system-site-packages
-	$(PIP) install --cert $(CERT) --upgrade pip
+	$(PYTHON) -m venv $(VENV) --prompt $(PENV) --system-site-packages
+	$(PIP) install --upgrade pip
 
 
-install-sphinx:
-	$(PIP) install --upgrade sphinx
+install-workflow-deps:
+	$(PIP) install -r $(WKFW)feature_demo/requirements.txt
 
 
 install-pip-mysql:
 	$(PIP) install -r requirements/mysql.txt
 
 
-install-tasks:
+install-merlin:
 	$(PIP) install -e .
 
 
-install-scipy:
-	$(PIP) install --cert $(CERT) scipy --ignore-installed
-
-
 # this only works outside the venv
 update: pull install clean
 
@@ -110,7 +103,6 @@ clean-py:
 # remove all studies/ directories
 clean-output:
 	-find $(MRLN) -name "studies*" -type d -exec rm -rf {} \;
-	-find workflows/ -name "studies*" -type d -exec rm -rf {} \;
 	-find . -maxdepth 1 -name "studies*" -type d -exec rm -rf {} \;
 	-find . -maxdepth 1 -name "merlin.log" -type f -exec rm -rf {} \;
 
@@ -156,8 +148,3 @@ check-camel-case: clean-py
 # run all checks
 checks: check-style check-camel-case
 
-
-# basic shortcut for starting celery workers
-start-workers:
-	celery worker -A merlin -l INFO
-