Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up sqlite database and replicate to Spaces #2

Open
AetherUnbound opened this issue Nov 6, 2022 · 3 comments
Open

Set up sqlite database and replicate to Spaces #2

AetherUnbound opened this issue Nov 6, 2022 · 3 comments

Comments

@AetherUnbound
Copy link
Contributor

We're hoping that the Airflow instance will be ephemeral and so we need a mechanism for managing data that needs to be backed up (e.g. the metadata database).

One quick way to do this to start would be use an SQLite database rather than Postgres, then set up an hourly DAG that copies the database file to Spaces. That would provide an easy recovery mechanism for us if we needed to tear down the instance.

Note that the official Airflow docs specify to never use sqlite for production, so ideally we'll have a better setup with PG WAL replication or something down the line (CC @mepholic @mpuckett159).

@mpuckett159
Copy link
Contributor

Imo it's fine to just use SQLite because this is like kinda production but not really. It's not that big a deal if it falls over and needs to be rebuilt again.

@AetherUnbound
Copy link
Contributor Author

Turns out that we can't use the LocalExecutor with sqlite, only the SequentialExecutor. This essentially means we can't have tasks run concurrently with sqlite, we'd need to have postgres in order to do that. Again I think that's fine to start, but postgres will be necessary down the line.

@AetherUnbound
Copy link
Contributor Author

Haha oh no
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 📋 Backlog
Development

No branches or pull requests

2 participants