Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epic/sof 2008 - ML POC, PD [DO NOT MERGE] #66

Open
wants to merge 83 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
737f2ae
drafted ML json schema and examples
Feb 28, 2017
42f6a38
method directory cleanup
Feb 28, 2017
c9d2293
feature/SOF-2002 added json schema for ML units and moved model param…
Feb 28, 2017
12304a2
Adjusted JSON Schema to include ML units
Mar 6, 2017
a94f27b
deleted old ML units
Mar 6, 2017
a4018d7
deleted old ML units example schema
Mar 6, 2017
5a9d5cf
Fixed JSON formatting errors
Mar 6, 2017
1d2f91a
modified model data
Mar 6, 2017
7459af6
Reorganized ML model and method schema
Mar 7, 2017
755b121
ML units cleanup
Mar 7, 2017
57d3a98
Adjusted JSON schema for ML units
Mar 8, 2017
089cb4a
Added examples for ML input
Mar 8, 2017
e48c227
fixed parameters input data
Mar 8, 2017
4d6e64d
Merge pull request #24 from Exabyte-io/feature/SOF-2002
Mar 8, 2017
ceae426
feature/SOF-2046 added bandgap_prediction_workflow json
Mar 8, 2017
df361fb
feature/SOF-2046 fixed ML workflow json
Mar 9, 2017
d5c391b
fixed ml workflow path and input parameters
Mar 9, 2017
79ddbb4
feature/SOF-2046 adjusted target porperty name and method parameters
Mar 9, 2017
aac4917
fixed ML input example json
Mar 9, 2017
7c07bc1
adjusted method parameters schema
Mar 9, 2017
e9a088e
adjusted method parameters json
Mar 9, 2017
ac9c866
Merge pull request #26 from Exabyte-io/feature/SOF-2046
Mar 9, 2017
1d7bfb1
adjusted ML input schema
Mar 16, 2017
397810b
modified ML POC input schema and example workflow
Mar 17, 2017
f077194
updated ML input JSON schema
Mar 17, 2017
766a89e
reorganized ML tasks and input schema
Mar 18, 2017
4ea0521
Re-added fit and predict keys
Mar 18, 2017
00e8f4b
feature/SOF-2075 modified ML examples to match current schema
Mar 18, 2017
fb500b0
Added exabyteId and changed result name to predict
timurbazhirov Mar 19, 2017
32d6347
Update input.json
timurbazhirov Mar 19, 2017
49465f3
Update input.json
timurbazhirov Mar 19, 2017
6530e11
Update input.json
timurbazhirov Mar 19, 2017
a4d7695
Merge pull request #28 from Exabyte-io/feature/SOF-2075
Mar 20, 2017
307f9f7
modified json schema to adjust inouts to predict unit
Mar 22, 2017
2af335c
modified json schema to adjust inouts to predict unit
Mar 22, 2017
6c9d006
modified ML input JSON schema
Mar 23, 2017
336c0a7
Merge pull request #33 from Exabyte-io/feature/SOF-1997
Mar 24, 2017
84ca7cd
Merge branch 'dev' into epic/SOF-2050
Mar 30, 2017
2d59167
added REST API configuration
mohammadidinani Apr 4, 2017
29ff241
added REST API credentials
mohammadidinani Apr 4, 2017
5394514
Merge pull request #38 from Exabyte-io/feature/SOF-2092
mohammadidinani Apr 5, 2017
b156325
added schema for material fingerprint primary descriptors
May 2, 2017
4859dd7
moved physical descriptors to elements schema
May 3, 2017
2f1b7b0
refactored JSON schema for material fingerprints
May 10, 2017
2c978f6
removed fingerprints from job and added to materials
May 10, 2017
43689b6
cleanup
May 10, 2017
c304ce6
Merge branch 'dev' into epic/SOF-2008
May 11, 2017
12894d8
refactored material fingerprint schema
May 13, 2017
425e471
cleanup and refactoring
May 13, 2017
9f7d83b
cleanup and refactoring
May 15, 2017
8fa82a7
Cleanups and renaming
timurbazhirov May 15, 2017
391d6f7
Merge branch 'feature/SOF-2168' into feature/SOF-2210
May 15, 2017
c171f7a
adjusted schema for local_environment
May 16, 2017
9b2de90
Merge branch 'epic/SOF-2050' into epic/SOF-2008
May 17, 2017
61235ec
Merge branch 'epic/SOF-2008' into feature/SOF-2289
May 17, 2017
bb68532
modified JSON schema to include elemental descriptors
May 19, 2017
5fc3d78
Adjusted naming of primary schemas
timurbazhirov May 20, 2017
d42df91
Merge branch 'epic/SOF-2008' into feature/SOF-2168
timurbazhirov May 22, 2017
15479a7
Merge pull request #44 from Exabyte-io/feature/SOF-2168
May 22, 2017
82c66e7
removed atomic category
May 24, 2017
bfc5e60
moved element schema to material/elements
May 24, 2017
81acc74
added required fields in JSON schema
May 24, 2017
e563618
Merge branch 'epic/SOF-2008' into feature/SOF-2291
May 25, 2017
829ec80
Merge pull request #50 from Exabyte-io/feature/SOF-2291
May 25, 2017
0a09489
files cleanup
May 25, 2017
c9a3124
Merge branch 'feature/SOF-2291' into epic/SOF-2008
May 25, 2017
d05d390
adjusted JSON schema for fit and predict methods with crossvalidation
May 29, 2017
4400ca6
Merge branch 'dev' into feature/SOF-2323
May 30, 2017
4ed44e0
Merge branch 'dev' into epic/SOF-2008
May 30, 2017
0c94be6
merged latest dev
May 30, 2017
34ea766
Merge branch 'epic/SOF-2008' into feature/SOF-2323
May 30, 2017
842204a
adjusted JSON schema for ML predict object
May 31, 2017
c2d9fa2
Added method/model aggregators
timurbazhirov May 31, 2017
8e9fda1
Reindented validator source
timurbazhirov May 31, 2017
c07413e
Refactored ml methods and units
timurbazhirov May 31, 2017
a0670ba
Fixed tests
timurbazhirov May 31, 2017
678ea28
fixed failed tests
May 31, 2017
ea234d4
Merge pull request #53 from Exabyte-io/feature/SOF-2340-TB
May 31, 2017
324bf3e
clean up and minor changes in ml input fit predict json
May 31, 2017
be03ac2
removed arguments from fit input
May 31, 2017
e5060d8
refactoring and added required properties
May 31, 2017
3949734
added arguments to input fit
Jun 7, 2017
8f7d89b
Merge pull request #52 from Exabyte-io/feature/SOF-2340
Jun 7, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions example/job.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
{
"status": "pre-submission",
"mode": "normal",
"_material": {
"...": "include(job/_material.json)"
},
"_materials": [
{
"...": "include(job/_material.json)"
}
],
"compute": {
"...": "include(job/compute.json)"
},
Expand All @@ -24,5 +26,8 @@
},
"_project": {
"...": "include(job/_project.json)"
},
"api": {
"...": "include(job/_api.json)"
}
}
4 changes: 4 additions & 0 deletions example/job/_api.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"userId": "jnLJpf9vJKYtFoQxc",
"authToken": "BvsTQ3xv-c4laO4Po3PQWsQty4GVwtDL_f2KXv6uFxz"
}
3 changes: 3 additions & 0 deletions example/job/input.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"...": "include(theory/ml/input.json)"
}
119 changes: 119 additions & 0 deletions example/job/software/ml/linear/workflow/fit_predict_band_gap.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
{
"_id": "ML bandgap prediction",
"title": "SiGe alloy bandgap prediction",
"titleSlug": "Template for ML POC SiGe alloy bandgap prediction",
"compute": {
"ppn": 1,
"queue": "OR",
"timeLimit": "1:00:00",
"notify": "n",
"nodes": 1,
"arguments": {}
},
"model": {
"method": {
"workflow": {
"name": "sige alloy bandgap prediciton using least squares regression",
"app": "machine_learning",
"units": [
{
"flowchartId": "1",
"head": true,
"type": "execution",
"name": "fit_MLmethod",
"next": "2",
"execution": {
"app": {
"name": "ml",
"exec": "rupy",
"version": "0.1.0"
},
"preProcessors": [],
"input": [
{
"fit": {
"type": "linear",
"subtype": "least_squares",
"arguments": {
"sampleWeight": true,
"trainDataSize": 0.75
}
},
"targetProperties": [
"band_gaps"
],
"features": [
"elemental_composition",
"density"
],
"exabyteIds": [
"3qp8j2CqCgsSa6ZKj",
"NdRNhkKWv4Xj8sxd8",
"AWLTt3qa5x6xRG353",
"cR2uFML56FL6rMf2v"
]
}
]
},
"results": [
{
"name": "predict"
}
]
},
{
"flowchartId": "2",
"type": "execution",
"name": "predict_property",
"head": false,
"execution": {
"app": {
"name": "ml",
"exec": "rupy",
"version": "0.1.0"
},
"preProcessors": [],
"input": [
{
"predict": {
"scope": "1",
"name": "predict"
},
"exabyteIds": [
"3qp8j2CqCgsSa6ZKj"
]
}
]
},
"results": [
{
"name": "band_gaps"
}
]
}
]
}
}
},
"_project": {
"_id": "56c569d7fe45004bec30b5e7",
"name": "Project 1",
"owner": {
"_id": "sdkucJLKbsaa2ajksh",
"name": "demo",
"type": 10
},
"slug": "project-1",
"goldName": "exabyte-project-1"
},
"version": "0.1.0",
"owner": {
"_id": "paxJukXpDBiSRjfRZ",
"name": "demo",
"type": 10
},
"creator": {
"username": "demo",
"_id": "paxJukXpDBiSRjfRZ"
}
}
9 changes: 9 additions & 0 deletions example/job/theory/ml/_input_fit.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"arguments": {
"...": "include(linear/input_fit.json)"
},
"method": {
"type": "linear",
"subtype": "least_squares"
}
}
8 changes: 8 additions & 0 deletions example/job/theory/ml/input.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"...": "include(input/predict.json)",
"materials": {
"exabyteIds": [
"LCthJ6E2QabYCZqf3"
]
}
}
11 changes: 11 additions & 0 deletions example/job/theory/ml/input/fit.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"...": "include(../_input_fit.json)",
"type": "fit",
"features": [
"elemental_composition",
"electronegativity"
],
"properties": [
"band_gaps"
]
}
4 changes: 4 additions & 0 deletions example/job/theory/ml/input/predict.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"type": "predict",
"method": "include(../linear.json)"
}
21 changes: 21 additions & 0 deletions example/job/theory/ml/linear.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"type": "linear",
"subtype": "least_squares",
"dataFrame": {
"...": "include(linear/data_frame.json)"
},
"data": {
"perProperty": [
{
"...": "include(linear/data.json)"
}
]
},
"precision": {
"perProperty": [
{
"...": "include(linear/precision.json)"
}
]
}
}
27 changes: 27 additions & 0 deletions example/job/theory/ml/linear/data.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"name": "band_gaps:indirect",
"coefficients": [
-3.339e-01,
2.555e-01
],
"features": [
"elemental_composition",
"electronegativity"
],
"intercept": [
0.5
],
"scores": [
0.018,
0.513
],
"scaler": [
{
"name": "mean",
"values": [
0.5,
1.95
]
}
]
}
16 changes: 16 additions & 0 deletions example/job/theory/ml/linear/data_frame.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"c8zG3XcnnTybfP3QQ": {
"electronegativity": 1.90,
"elemental_composition": {
"Ge": 0.3125,
"Si": 0.6875
}
},
"wWK2XCyivQrG8j34Z": {
"electronegativity": 2.00,
"elemental_composition": {
"Ge": 0.1875,
"Si": 0.8125
}
}
}
6 changes: 6 additions & 0 deletions example/job/theory/ml/linear/input_fit.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"sampleWeight": true,
"numFeatures": 2,
"scaler": "StandardScaler",
"numSplits":3
}
6 changes: 6 additions & 0 deletions example/job/theory/ml/linear/precision.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"name": "band_gaps:indirect",
"trainingError": 0.1,
"cvError": 0.3,
"score": 0.8
}
3 changes: 1 addition & 2 deletions example/job/workflow/unit/execution.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@
"compute": null,
"input": [
{
"content": "K_POINTS automatic\n2 2 2 0 0 0",
"name": "pw_scf.in"
"...": "include(execution/input.json)"
}
]
}
4 changes: 4 additions & 0 deletions example/job/workflow/unit/execution/input.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"content": "K_POINTS automatic\n2 2 2 0 0 0",
"name": "pw_scf.in"
}
4 changes: 4 additions & 0 deletions example/material/elements/_elemental_data.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"name": "valence_electrons",
"value": 4
}
41 changes: 41 additions & 0 deletions example/material/elements/element.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
{
"element": "Si",
"properties": [
{
"name": "valence_electrons",
"value": 4
},
{
"name": "unpaired_electrons",
"value": 4
},
{
"name": "atomic_radius",
"value": 111.0,
"units": "pm"
},
{
"name": "electron_configuration",
"value": "[Ne]3s23p2"
},
{
"name": "electronegativity",
"value": 1.90
},
{
"name": "ionization_potential",
"value": 8.15,
"units": "eV"
},
{
"name": "density",
"value": 2.32,
"units": "g/cm^3"
},
{
"name": "cohesive_energy",
"value": 4.63,
"units": "eV/atom"
}
]
}
17 changes: 17 additions & 0 deletions example/material/fingerprint.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"elemental":[
{
"...": "include(fingerprint/elemental.json)"
}
],
"primary": [
{
"...": "include(fingerprint/primary.json)"
}
],
"secondary": [
{
"...": "include(characteristic.json)"
}
]
}
6 changes: 6 additions & 0 deletions example/material/fingerprint/elemental.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"type": "element-specific",
"subtype": "numeric",
"name": "ionization_potential",
"...": "include(elemental/ionization_potential.json)"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"...": "include(../schemas/element_specific.json)",
"name": "ionization_potential",
"units": "eV"
}
7 changes: 7 additions & 0 deletions example/material/fingerprint/primary.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"type": "compound-specific-array",
"subtype": "numeric",
"category": "stoichiometric",
"name": "p_norms",
"...": "include(primary/p_norms.json)"
}
3 changes: 3 additions & 0 deletions example/material/fingerprint/primary/p_norms.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"...": "include(../schemas/compound_specific_array.json)"
}
21 changes: 21 additions & 0 deletions example/material/fingerprint/schemas/compound_specific_array.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "p_norms",
"data": [
{
"name": "p03",
"value": 0.693
},
{
"name": "p05",
"value": 0.671
},
{
"name": "p07",
"value": 0.667
},
{
"name": "p09",
"value": 0.667
}
]
}
Loading