OpenFn/Lightning (alpha)

Lightning extends the existing OpenFn Digital Public Good, providing a web UI to visually manage and execute complex workflow automation and data integration projects. Learn more about the entire free and open source OpenFn Integration Toolkit toolkit at docs.openfn.org.

Getting Started

• If you only want to RUN Lightning on your own server, we recommend using Docker.
• If you want to DEPLOY Lightning, we recommend Docker builds and Kubernetes.
• If you want to CONTRIBUTE to the project, we recommend setting up Elixir on your local machine.

Run via Docker

1. Install the latest version of Docker
2. Clone the repo using git
3. Copy the .env.example file to .env
4. Run docker compose run --rm web mix ecto.migrate

By default the application will be running at localhost:4000.

You can then rebuild and run with docker compose build and docker compose up. See "Problems with Docker" for additional troubleshooting help. Note that you can also create your own docker-compose.yml file, configuring a postgres database and using a pre-built image from Dockerhub.

Deploy on external infrastructure

See Deployment for more detailed information.

Contribute to this project

First, thanks for being here! You're contributing to a digital public good that will always be free and open source and aimed at serving innovative NGOs, governments, and social impact organizations the world over! You rock. ❤️

FYI, Lightning is built in Elixir, harnessing the Phoenix Framework. Currently, the only unbundled dependency is a PostgreSQL database.

Set up your environment

If you have push access to this repository (are you an authorized maintainer?) then you'll be able to make changes and push them to a feature branch before submitting a pull request. If you're new to OpenFn, you'll need to make a fork and push your changes there.

Once you're ready to submit a pull request, you can click the "compare across forks" link on GitHub's pull request interface and then open one for review.

Clone the repo and set ENVs

git clone [email protected]:OpenFn/Lightning.git # or from YOUR fork!
cd Lightning
cp .env.example .env # and adjust as necessary!

Take note of database names and ports in particular—they've got to match across your Postgres setup and your ENVs.

Database Setup

If you're already using Postgres locally, create a new database called lightning_dev, for example.

If you'd rather use Docker to set up a Postgres DB, create a new volume and image:

docker volume create lightning-postgres-data

docker create \
  --name lightning-postgres \
  --mount source=lightning-postgres-data,target=/var/lib/postgresql/data \
  --publish 5432:5432 \
  -e POSTGRES_PASSWORD=postgres \
  postgres:14.1-alpine

docker start lightning-postgres

Elixir & Ecto Setup

We use asdf to configure our local environments. Included in the repo is a .tool-versions file that is read by asdf in order to dynamically make the specified versions of Elixir and Erlang available.

asdf install  # Install language versions
mix local.hex
mix deps.get
mix local.rebar --force
mix ecto.create # Create a development database in Postgres
mix ecto.migrate
mix openfn.install.runtime
npm install --prefix assets

Run the app

Lightning is a web app. To run it in interactive Elixir mode, start the development server by running with your environment variables by running:

env $(cat .env | grep -v "#" | xargs ) iex -S mix phx.server

Once the server has started, head to localhost:4000 in your browser.

Run the tests

Before the first time running the tests, you need a test database setup.

MIX_ENV=test mix ecto.create

And then after that run the tests using:

MIX_ENV=test mix test

We also have test.watch installed which can be used to rerun the tests on file changes.

Security and Standards

We use a host of common Elixir static analysis tools to help us avoid common pitfalls and make sure we keep everything clean and consistent.

In addition to our test suite, you can run the following commands:

• mix format --check-formatted
  Code formatting checker, run again without the --check-formatted flag to have your code automatically changed.
• mix dialyzer
  Static analysis for type mismatches and other common warnings. See dialyxir.
• mix credo
  Static analysis for consistency, and coding standards. See Credo.
• mix sobelow
  Check for commonly known security exploits. See Sobelow.
• MIX_ENV=test mix coveralls
  Test coverage reporter. This command also runs the test suite, and can be used in place of mix test when checking everything before pushing your code. See excoveralls.

For convenience there is a verify mix task that runs all of the above and defaults the MIX_ENV to test.

Generating Documentation

You can generate the HTML and EPUB documentation locally using:

mix docs and opening doc/index.html in your browser.

Troubleshooting

Problems with Docker

Versions

The build may not work on old versions of Docker and Docker compose. It has been tested against:

Docker version 20.10.17, build 100c701
Docker Compose version v2.6.0

Starting from scratch

If you're actively working with docker, you start experiencing issues, and you would like to start from scratch you can clean up everything and start over like this:

# To remove any ignored files and reset your .env to it's example
git clean -fdx && cp .env.example .env
# You can skip the line below if you want to keep your database
docker compose down --rmi all --volumes

docker compose build --no-cache web && \
  docker compose create --force-recreate

docker compose run --rm web mix ecto.migrate
docker compose up