Skip to content

Commit

Permalink
Workflows Community Initiative Metadata (#355)
Browse files Browse the repository at this point in the history
* Added Workflows Community Initiative metadata info; fixed some old links

* Run black
  • Loading branch information
lucpeterson authored Feb 3, 2022
1 parent 75dde86 commit 05cdecb
Show file tree
Hide file tree
Showing 11 changed files with 28 additions and 12 deletions.
12 changes: 12 additions & 0 deletions .wci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
name: Merlin

icon: https://raw.githubusercontent.com/LLNL/merlin/main/docs/images/merlin_icon.png

headline: Enabling Machine Learning HPC Workflows

description: The Merlin workflow framework targets large-scale scientific machine learning (ML) workflows in High Performance Computing (HPC) environments. Merlin is a producer-consumer workflow model that enables multi-machine, cross-batch job, dynamically allocated yet persistent workflows capable of utilizing surge-compute resources. Key features are a flexible and intuitive HPC-centric interface, low per-task overhead, multi-tiered fault recovery, and a hierarchical sampling algorithm that allows for highly scalable task execution and queuing to ensembles of millions of tasks.

documentation:
general: https://merlin.readthedocs.io/
installation: https://merlin.readthedocs.io/en/latest/modules/installation/installation.html
tutorial: https://merlin.readthedocs.io/en/latest/tutorial.html
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- Auto-release of pypi packages
- Workflows Community Initiative metadata file

### Fixed
- Old references to stale branches

## [1.8.3]
### Added
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
[![Issues](https://img.shields.io/github/issues/LLNL/merlin)](https://github.com/LLNL/merlin/issues)
[![Pull requests](https://img.shields.io/github/issues-pr/LLNL/merlin)](https://github.com/LLNL/merlin/pulls)

![Merlin](https://raw.githubusercontent.com/LLNL/merlin/master/docs/images/merlin.png)
![Merlin](https://raw.githubusercontent.com/LLNL/merlin/main/docs/images/merlin.png)

## A brief introduction to Merlin
Merlin is a tool for running machine learning based workflows. The goal of
Expand Down Expand Up @@ -133,6 +133,6 @@ the Merlin community, you agree to abide by its rules.


## License
Merlin is distributed under the terms of the [MIT LICENSE](https://github.com/LLNL/merlin/blob/master/LICENSE).
Merlin is distributed under the terms of the [MIT LICENSE](https://github.com/LLNL/merlin/blob/main/LICENSE).

LLNL-CODE-797170
Binary file added docs/images/merlin_icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/source/modules/contribute.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,6 @@ Merlin is an open source project, so contributions are welcome. Contributions ca

Contributing to Merlin is easy! Just `send us a pull request <https://github.com/LLNL/merlin/pulls>`_ from your fork. Before you send it, summarize your change in the ``[Unreleased]`` section of ``CHANGELOG.md`` and make sure develop is the destination branch. We also appreciate `squash commits <https://github.com/LLNL/merlin/wiki/Squash-commits>`_ before pull requests are merged.

Merlin uses a rough approximation of the Git Flow branching model. The develop branch contains the latest contributions, and master is always tagged and points to the latest stable release.
Merlin uses a rough approximation of the Git Flow branching model. The develop branch contains the latest contributions, and main is always tagged and points to the latest stable release.

If you're a contributor, try to test and run on develop. That's where all the magic is happening (and where we hope bugs stop).
2 changes: 1 addition & 1 deletion merlin/examples/workflows/feature_demo/scripts/pgen.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ def get_custom_generator(env, **kwargs):
p_gen = ParameterGenerator()
params = {
"X2": {"values": [1 / i for i in range(3, 6)], "label": "X2.%%"},
"N_NEW": {"values": [2 ** i for i in range(1, 4)], "label": "N_NEW.%%"},
"N_NEW": {"values": [2**i for i in range(1, 4)], "label": "N_NEW.%%"},
}

for key, value in params.items():
Expand Down
4 changes: 2 additions & 2 deletions merlin/examples/workflows/null_spec/scripts/launch_jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@

# launch n_samples * n_conc merlin workflow jobs
submit_path: str = os.path.abspath(os.path.dirname(os.path.abspath(__file__)))
concurrencies: List[int] = [2 ** 0, 2 ** 1, 2 ** 2, 2 ** 3, 2 ** 4, 2 ** 5, 2 ** 6]
samples: List[int] = [10 ** 1, 10 ** 2, 10 ** 3, 10 ** 4, 10 ** 5]
concurrencies: List[int] = [2**0, 2**1, 2**2, 2**3, 2**4, 2**5, 2**6]
samples: List[int] = [10**1, 10**2, 10**3, 10**4, 10**5]
nodes: List = []
c: int
for c in concurrencies:
Expand Down
2 changes: 1 addition & 1 deletion merlin/examples/workflows/openfoam_wf/scripts/learn.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
U = outputs["arr_0"]
enstrophy = outputs["arr_1"]

energy_byhand = np.sum(np.sum(U ** 2, axis=3), axis=2) / U.shape[2] / 2
energy_byhand = np.sum(np.sum(U**2, axis=3), axis=2) / U.shape[2] / 2
enstrophy_all = np.sum(enstrophy, axis=2)

X = np.load(inputs_dir + "/samples.npy")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
U = outputs["arr_0"]
enstrophy = outputs["arr_1"]

energy_byhand = np.sum(np.sum(U ** 2, axis=3), axis=2) / U.shape[2] / 2
energy_byhand = np.sum(np.sum(U**2, axis=3), axis=2) / U.shape[2] / 2
enstrophy_all = np.sum(enstrophy, axis=2)

X = np.load(inputs_dir + "/samples.npy")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,22 +17,22 @@ def rosenbrock(X):
def rastrigin(X, A=10):
first_term = A * len(inputs)

return first_term + sum([(x ** 2 - A * np.cos(2 * math.pi * x)) for x in X])
return first_term + sum([(x**2 - A * np.cos(2 * math.pi * x)) for x in X])


def ackley(X):
firstSum = 0.0
secondSum = 0.0
for x in X:
firstSum += x ** 2.0
firstSum += x**2.0
secondSum += np.cos(2.0 * np.pi * x)
n = float(len(X))

return -20.0 * np.exp(-0.2 * np.sqrt(firstSum / n)) - np.exp(secondSum / n) + 20 + np.e


def griewank(X):
term_1 = (1.0 / 4000.0) * sum(X ** 2)
term_1 = (1.0 / 4000.0) * sum(X**2)
term_2 = 1.0
for i, x in enumerate(X):
term_2 *= np.cos(x) / np.sqrt(i + 1)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ def get_custom_generator(env, **kwargs) -> ParameterGenerator:
p_gen: ParameterGenerator = ParameterGenerator()
params: Dict[str, Union[List[float], str]] = {
"X2": {"values": [1 / i for i in range(3, 6)], "label": "X2.%%"},
"N_NEW": {"values": [2 ** i for i in range(1, 4)], "label": "N_NEW.%%"},
"N_NEW": {"values": [2**i for i in range(1, 4)], "label": "N_NEW.%%"},
}
key: str
value: Union[List[float], str]
Expand Down

0 comments on commit 05cdecb

Please sign in to comment.