Skip to content

Commit

Permalink
readme - added results to examples
Browse files Browse the repository at this point in the history
  • Loading branch information
crispyDyne committed Jul 29, 2020
1 parent 6db2d55 commit 644bd12
Show file tree
Hide file tree
Showing 3 changed files with 54 additions and 20 deletions.
10 changes: 4 additions & 6 deletions examples/examples.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,10 @@

from rq_manager import manager, getProjectResults
from exampleTasks import addSubJob, simpleTask, addJobs, addSubJob

import json
from pprint import pprint

def jsonCopy(d):
return json.loads(json.dumps(d))

#### Examples are slow because "sleepTime" default is 1 second ####

# Tell RQ what Redis connection to use
redis_conn = Redis()
Expand Down Expand Up @@ -69,7 +67,7 @@ def jsonCopy(d):
'jobs':[ # these two jobs will be run first
{'func':simpleTask,'args': 1},
{'func':addJobs,'args': 2} # This job adds new jobs
# New Jobs are placed here
# New jobs are placed here
],
},
{'func':simpleTask,'args': 3}]
Expand All @@ -86,7 +84,7 @@ def jsonCopy(d):
'jobs':[ # these two jobs will be run first
{'func':simpleTask,'args': 1},
{'func':addSubJob,'args': 2} # This job adds a new job with child jobs
# {'jobs': New Jobs are placed here}
# {'jobs': New child jobs are placed here}
],
},
{'func':simpleTask,'args': 2}]
Expand Down
62 changes: 48 additions & 14 deletions readme.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@

## rq-manager
Manage jobs with multiple or tree-like dependancy.
Manage rq jobs with multiple or tree-like dependancy.

Create a "project", which is a tree of jobs, then give it to the "manager" to complete it!

```python
q = Queue()
managerJob = q.enque(manager,project)
projectResults = getProjectResults(managerJob)
```

## A few examples
Define a simple job:
Expand All @@ -22,6 +25,11 @@ project = {'jobs':[
{'func':simpleTask,'args': 2}]
}
```
returns
```python
[2, 4]

```

### Run jobs in series:
If a job is marked as blocking, the following jobs will wait to run.
Expand All @@ -31,30 +39,46 @@ project = {'jobs':[
{'func':simpleTask,'args': 2}]
}
```
returns
```python
[2, 4]

```

### Run with dependent arguments:
A job can use the the results of a previous job as its inputs. It will wait for the previous job to finish, but it will not block later jobs.
```Python
project = {'jobs':[
{'func':simpleTask,'args': 1},
{'func':simpleTask, 'previousJobArgs': True}, # this job will wait to start until the previous job is finished
{'func':simpleTask, 'previousJobArgs': True}, # this job will wait
{'func':simpleTask,'args': 3}] # this job will NOT wait
}
```
returns
```python
[2, 4, 6]

```

### Run jobs with multiple dependancy:
A job can have child jobs. The parent job is not finised until all of the child jobs are finished.
```Python
project = {'jobs':[
{
'blocking':True # this job, and its child jobs, must finish before moving on.
'blocking':True, # this job, and its child jobs, must finished before moving on.
'jobs':[
{'func':simpleJob,'args': 1},
{'func':simpleJob,'args': 2}],
{'func':simpleTask,'args': 1},
{'func':simpleTask,'args': 2}],
},
{ # this job will only run when the blocking job above finishes.
'func':simpleJob,'args': 2}]
'func':simpleTask,'args': 3
}
]}
```
returns
```python
[[2, 4], 6]

```

### Add jobs as you go
Expand All @@ -72,15 +96,20 @@ Jobs will be appended to the current job array
project = {'jobs':[
{
'blocking':True,
'jobs':[ # these two jobs will run first
{'func':simpleTask,'args': 2},
{'func':addJobs,'args': 4} # This job adds new jobs
'jobs':[ # these two jobs will be run first
{'func':simpleTask,'args': 1},
{'func':addJobs,'args': 2} # This job adds new jobs
# New Jobs are placed here
],
},
{'func':simpleTask,'args': 2}]
{'func':simpleTask,'args': 3}]
}
```
returns
```python
[[2, 4, 8, 10], 6]

```

### Add job with child jobs as you go
Define a job that creates new a job filled with child jobs.
Expand All @@ -93,20 +122,25 @@ def addSubJob(n):
newSubJob = {'jobs':newJobArray }
return {'result':2*n, 'addJobs':newSubJob}
```
A new sub job with subjobs will
A sub job will be added with child jobs.
```Python
project = {'jobs':[
{
'blocking':True,
'jobs':[ # these two jobs will be run first
{'func':simpleTask,'args': 2},
{'func':addSubJob,'args': 4} # This job adds a new job with child jobs
# {'jobs': New Jobs are placed here}
{'func':simpleTask,'args': 1},
{'func':addSubJob,'args': 2} # This job adds a sub job with child jobs
# {'jobs': New child jobs are placed here}
],
},
{'func':simpleTask,'args': 2}]
}
```
returns
```python
[[2, 4, [8, 10]], 6]

```

## Redis Help
Install Redis Server
Expand Down
2 changes: 2 additions & 0 deletions rq_manager/manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,8 @@ def getJobResults(job):


def getProjectResults(managerJob):
# Get results of the project.
# This function is blocking, and only returns when the project is complete
project = None
finished = False
while not finished:
Expand Down

0 comments on commit 644bd12

Please sign in to comment.