-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Job Spec #8
Draft
expede
wants to merge
42
commits into
main
Choose a base branch
from
initial-job-spec
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Job Spec #8
Changes from 5 commits
Commits
Show all changes
42 commits
Select commit
Hold shift + click to select a range
e965fed
Let's get this party started
expede 2ea97a6
Typo
expede a2b337f
WIP -- mocking out the format
expede 0eed3f8
Iterating on lots of small details, trying to cover Bacalhau
expede 238711a
Trying another approach for pipelining
expede 1eb8ab6
Flattening the structure, did some more prototyping, started on text
expede ec8b020
Slowly discovering the shape of the spec
expede 4c70aab
Simplifying!
expede 2dbdb28
Formalizing
expede bd5bde2
Start skwtching out WarpForge
expede 35d1bb2
Fleshing out effects and intro sections
expede 933dfc4
Saving WIP
expede 024e952
Flesh out pipeling, rename several fields (e.g. run)
expede ab7961a
Start to IPLDify
expede ab95752
job -> workflow, break out task spec
expede 27e75c1
Task spec
expede ba54b40
Spec collectio table
expede 7d7073f
Add subspecs / roadmap
expede 84c71a4
Mocking up a (hopefully) better Docker
expede 31b1142
CHa!
expede 5acd816
Start on invocations (stage after job request)
expede be735ba
Forgot to push last week
expede 089ef46
Explicitely link out to UCAN invcoations/actions
expede 2436eee
Envelope type
expede 5e67ddd
Start tightening up and splitting out stuff that's not diretly about …
expede 1cc8b18
Simplify down Task, break out effect, rely more heavily on UCAN Invoc…
expede c85da36
Laying out new spec
expede 14e23e2
Remove the "provisionally"
expede 2922260
Units, early cleanup
expede b0aa0d3
Tightening up the types; nesting instead of subtyppin the config
expede 9123050
Tightening up the wrapper format
expede cc4d1df
More tighteninhg up of just the workflwos
expede 0bfafca
Change based on clarifications about IPLD Schema from rvagg
expede e1578b4
Add links
expede 4b93f49
Really really getting quite close! Getting late... must sleep
expede ef904a1
Catch!
expede a3de19d
Many thank-yous
expede 167250d
Fix many but not all spelling issues
expede e6ec07a
Rewire on catch
expede b40ef5d
Eliminate infinite loop
expede 22a827f
Wrapping up cleanup
expede 9572701
Start cutting cruft
expede File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
# IPVM Job Spec | ||
|
||
## Editors | ||
|
||
* [Brooklyn Zelenka](https://github.com/expede), [Fission](https://fission.codes) | ||
|
||
## Authors | ||
|
||
## Language | ||
|
||
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119](https://datatracker.ietf.org/doc/html/rfc2119). | ||
|
||
# 0 Abstract | ||
|
||
|
||
# 1 Motivation | ||
|
||
|
||
|
||
# 2 IPLD | ||
|
||
|
||
# 3 Acknowledgments | ||
|
||
|
||
# 4 Prior Art | ||
|
||
* [Docker Job Controller](https://kubernetes.io/docs/concepts/workloads/controllers/job/) | ||
* BucketVM (UCAN Invocation) | ||
* [WarpForge "Formula" v1](https://github.com/warpfork/warpforge/blob/master/examples/110-formula-usage/example-formula-exec.md) | ||
* [Bacalhau Job Spec](https://github.com/filecoin-project/bacalhau/blob/8568239299b5881bc90e3d6be2c9aa06c0cb3936/pkg/model/job.go#L192) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
digraph G { | ||
Pipeline -> {Job, JobL, JobR, JobE} | ||
|
||
// | ||
|
||
Job [shape = box] | ||
Job -> {Invocation, Config, Version, requestorDid} | ||
|
||
Invocation [shape = box] | ||
Invocation -> Wasm [label = "exec"] | ||
Invocation -> {a, b, c} [label = "arg"] | ||
|
||
// | ||
|
||
JobL [shape = box] | ||
JobL -> {InvocationL, ConfigL, VersionL, requestorDidL} | ||
|
||
InvocationL [shape = box] | ||
InvocationL -> WasmL [label = "exec"] | ||
InvocationL -> {d, e, f} [label = "arg"] | ||
|
||
// | ||
|
||
JobR [shape = box] | ||
JobR -> {InvocationR, ConfigR, VersionR, requestorDidR} | ||
|
||
InvocationR [shape = box] | ||
InvocationR -> WasmR [label = "exec"] | ||
InvocationR -> {g, h, i} [label = "arg"] | ||
|
||
// | ||
|
||
JobE [shape = box] | ||
JobE -> {InvocationE, ConfigE, VersionE, requestorDidE} | ||
|
||
InvocationE [shape = box] | ||
InvocationE -> WasmR [label = "exec"] | ||
InvocationE -> {j, k, l} [label = "arg"] | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
{ | ||
"type": "ipvm/job", | ||
"version": "0.0.1", | ||
"requestor": "did:key:zAlice", | ||
"nonce": "ABCDEF", | ||
"config": { | ||
"label": "fission/run_the_reports", | ||
"start": "asap", | ||
"timeoutAt": 1667878395, | ||
"maxGas": 4096, | ||
"authz": ["QmUcan1", "QmUcan2"], // TODO move to post-negotiated? | ||
"visibility": "public", | ||
"verification": { | ||
"method": "ipvm/optimistic-zk", | ||
"min": 1, | ||
"replication": 2, | ||
"referee": "ipns://abcdefghi" | ||
} | ||
}, | ||
"run": { | ||
"wasm": "bafyWasm", | ||
"args": [ | ||
{ "w": "Qm123456" }, | ||
{ "x": "Qmabcdef" }, | ||
{ "y": { "with": "dnslink://example.com", "run": "dnslink/resolve" } } | ||
{ "z": "QmFooBar" }, | ||
], | ||
"affinities": [], // TODO lives here, or in | ||
}, | ||
"before": { | ||
"database": { | ||
"with": "dnslink://example.com", | ||
"run": "dnslink/resolve" | ||
} | ||
}, | ||
"then": { | ||
"effects": { | ||
"bell": { | ||
"run": "system/bell", | ||
"on": "always" | ||
} | ||
}, | ||
"jobs": { | ||
"left": { | ||
"run": { | ||
"wasm": "bafyWasmLeft", | ||
"args": ["bafySomeArg"] | ||
}, | ||
"then": { | ||
"jobs": { | ||
"subleft": { | ||
"run": { | ||
"wasm": "bafyWasmSubLeft", | ||
"args": | ||
} | ||
} | ||
} | ||
} | ||
}, | ||
"right": { | ||
"run": "b" | ||
}, | ||
"end": { | ||
"run": { | ||
"wasm": "bafyWasmEnd", | ||
"input": { | ||
"foo": "jobs/left/subleft/" | ||
} | ||
} | ||
} | ||
}, | ||
"signature": "abcdef" | ||
} | ||
|
||
{ | ||
"type": "ipvm/deal", | ||
"version": "0.0.1", | ||
"accepter": "", | ||
"job": "Qmabcdef" | ||
} |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
{ | ||
"type": "ipvm/job", | ||
"version": "0.0.1", | ||
"requestor": "did:key:zAlice", | ||
"nonce": "ABCDEF", | ||
"config": { | ||
"label": "fission/run_the_reports", | ||
"start": "asap", | ||
"timeoutAt": 1667878395, | ||
"maxGas": 4096, | ||
"authz": ["QmUcan1", "QmUcan2"], // TODO move to post-negotiated? | ||
"visibility": "public", | ||
"verification": { | ||
"method": "ipvm/optimistic-zk", | ||
"min": 1, | ||
"replication": 2, | ||
"referee": "ipns://abcdefghi" | ||
} | ||
}, | ||
"jobs": { | ||
"database": { | ||
"run": { | ||
"effect": { | ||
"with": "dnslink://example.com", | ||
"do": "dnslink/resolve", | ||
"timeout": "30s" | ||
} | ||
} | ||
}, | ||
"start": { | ||
"run": { | ||
"wasm": "bafyWasm", | ||
"input": [ | ||
{ "w": "Qm123456" }, | ||
{ "x": "Qmabcdef" }, | ||
{ "y": { "from": "database", "out": 0 } } // "Let the dataflow through you" | ||
{ "z": "QmFooBar" }, | ||
], | ||
"affinities": [], // TODO lives here, or in | ||
}, | ||
"after": { | ||
"effects": { | ||
"bell": { | ||
"run": "system/bell", | ||
"when": "always" | ||
} | ||
} | ||
} | ||
}, | ||
"left": { | ||
"run": { | ||
"wasm": "bafyWasmLeft", | ||
"input": ["bafySomeArg"] | ||
} | ||
}, | ||
"subleft": { | ||
"run": { | ||
"wasm": "bafyWasmSubLeft", | ||
"input": [101, {"from": "left"}, 42] | ||
} | ||
}, | ||
"right": { | ||
"run": { | ||
"wasm": "bafyWasmRight", | ||
"input": [{"from": "start", "output": 4}, {"from": "database"}] | ||
} | ||
}, | ||
"end": { | ||
"run": { | ||
"wasm": "bafyWasmEnd", | ||
"input": { | ||
"foo": "jobs/left/subleft/" | ||
} | ||
} | ||
} | ||
} | ||
"signature": "abcdef" | ||
} | ||
|
||
{ | ||
"type": "ipvm/deal", | ||
"version": "0.0.1", | ||
"accepter": "", | ||
"job": "Qmabcdef" | ||
} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is the output here just "returning a number"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
4
? Current theory is that it'll be the index on the multivalued return in this completely made up example.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, the
4
. So,it's the index, gotcha.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Evidently the field name should be changed for clarity. Good feedback :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suggest named slots for clarity (and robustness to changes in # of output parameters)
Another suggestion for robustness: Optional expected type/schema for both input and outputs. Feeling there's a good opportunity for a Cambrian-like data lens structure here...
Maybe that's out of scope (because we have clear expectations of stability from the model) but then again, given that mutability is expected...