Skip to content

An implementation of GA4GH Workflow Execution Service Standard as a microservice.

License

Notifications You must be signed in to change notification settings

suecharo/genpei

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ed26262 · May 7, 2020

History

45 Commits
May 7, 2020
May 7, 2020
May 3, 2020
May 7, 2020
May 7, 2020
May 7, 2020
May 7, 2020
May 2, 2020
May 3, 2020
May 7, 2020
May 7, 2020
May 7, 2020
May 7, 2020
May 7, 2020
May 7, 2020
May 7, 2020

Repository files navigation

Genpei (源平)

pytest flake8 isort mypy

Japanese Document

Genpei (源平) is a standard implementation conforming to the Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API specification. A simple and highly scalable REST API Server using Flask and cwltool that follows the philosophy of Microservice. It supports the execution and management of Workflow written in Common Workflow Language (CWL).

Install and Run

Python 3.6 or higher.

$ pip3 install genpei
$ genpei

Docker

We also expect to run using Docker. Because of the compatibility of cwltool and Docker-in-Docker (DinD), you have to mount docker.dock, /tmp, etc. Please check the documentation in DockerHub - cwltool for more information.

$ docker-compose up -d --build
$ docker-compose exec app genpei

Usage

As API specifications, please check GitHub - GA4GH WES and SwaggerUI - GA4GH WES.

As the simplest example of a REST API Request, here is the result of a GET /service-info.

$ curl -X GET localhost:8080/service-info
{
  "auth_instructions_url": "https://github.com/suecharo/genpei",
  "contact_info_url": "https://github.com/suecharo/genpei",
  "default_workflow_engine_parameters": [],
  "supported_filesystem_protocols": [
    "http",
    "https",
    "file"
  ],
  "supported_wes_versions": [
    "1.0.0"
  ],
  "system_state_counts": {},
  "tags": {
    "wes_name": "genpei"
  },
  "workflow_engine_versions": {
    "cwltool": "3.0.20200324120055"
  },
  "workflow_type_versions": {
    "CWL": {
      "workflow_type_version": [
        "v1.0",
        "v1.1",
        "v1.1.0-dev1",
        "v1.2.0-dev1",
        "v1.2.0-dev2"
      ]
    }
  }
}

Startup Option (--host and --port) of Genpei allows you to change the startup host and port.

$ genpei --help
usage: genpei [-h] [--host HOST] [-p PORT] [--debug] [-r RUN_DIR]

Implementation of GA4GH WES OpenAPI specification using cwltool.

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           Host address of Flask. (default: 127.0.0.1)
  -p PORT, --port PORT  Port of Flask. (default: 8080)
  --debug               Enable debug mode of Flask.
  -r RUN_DIR, --run-dir RUN_DIR
                        Specify the run dir. (default: ./run)

$ genpei --host 0.0.0.0 --port 5000

Genpei manages submitted workflow files, workflow parameters, output files, etc. on the FileSystem. It is called Run dir. The location of default is ./run. You can change the location of Run dir by specifying the startup option (-r).

The run dir structure looks like the following. There are various files related to each run. Initialization and deletion of each run can be done by physical deletion using rm.

$ tree run
.
├── 11
│   └── 11a23a68-a914-427a-80cd-9ad6f7cfd256
│       ├── cmd.txt
│       ├── end_time.txt
│       ├── exe
│       │   ├── ERR034597_1.small.fq.gz
│       │   ├── ERR034597_2.small.fq.gz
│       │   ├── fastqc.cwl
│       │   ├── trimming_and_qc.cwl
│       │   ├── trimmomatic_pe.cwl
│       │   └── workflow_params.json
│       ├── exit_code.txt
│       ├── output
│       │   ├── ERR034597_1.small_fastqc.html
│       │   ├── ERR034597_1.small_fastqc.html_2
│       │   ├── ERR034597_1.small.fq.trimmed.1P.fq
│       │   ├── ERR034597_1.small.fq.trimmed.1U.fq
│       │   ├── ERR034597_1.small.fq.trimmed.2P.fq
│       │   └── ERR034597_1.small.fq.trimmed.2U.fq
│       ├── run.pid
│       ├── run_request.json
│       ├── start_time.txt
│       ├── state.txt
│       ├── stderr.log
│       └── stdout.log
├── 14
│   └── ...
├── 2d
│   └── ...
└── 6b
    └── ...

The execution of POST /runs is very complex. Examples using Python's requests are provided by GitHub - genpei/tests/post_runs_examples. Please use this as a reference

Development

The development environment starts with the following.

$ docker-compose -f docker-compose.dev.yml up -d --build
$ docker-compose -f docker-compose.dev.yml exec app bash

We use flake8, isort, and mypy as the Linter.

$
$ bash ./tests/lint_and_style_check/flake8.sh
$ bash ./tests/lint_and_style_check/isort.sh
$ bash ./tests/lint_and_style_check/mypy.sh

We use pytest as a Test Tool.

$ pytest .

Lisense

Apache-2.0. See the LICENSE.