COVID AMP API server (https://api.covidamp.org/docs)

Application programming interface (API) server for the COVID Policy Tracker website. Note: This codebase also includes data ingest code. See COVID AMP data update brief checklist for more information after reading this README.

A list of all relevant web resources for this project follows.

• https://covidamp.org. Distribution URL for main COVID AMP site. By default, uses API server at https://api.covidamp.org. See repository covid-npi-policy/tree/master for frontend code.
• https://test.covidamp.org. Distribution URL for main COVID AMP site. By default, uses API server at https://api-test.covidamp.org. See repository covid-npi-policy/tree/dev for frontend code.
• https://api.covidamp.org/docs. Interactive documentation for main COVID AMP API server. By default, connects to database covid-npi-policy at AWS RDS host talus-prod. Corresponds to Elastic Beanstalk environment amp-prod2.
• https://api-test.covidamp.org/docs. Interactive documentation for test COVID AMP API server. By default, connects to database covid-npi-policy-test at AWS RDS host talus-prod. Corresponds to Elastic Beanstalk environment amp-dev2.
• https://airtable.com/tblLpBz6sEExhYVVC. COVID AMP main dataset on Airtable. All policy data are ingested from here.
• https://s3.console.aws.amazon.com/s3/buckets/covid-npi-policy-files. COVID AMP storage S3 bucket containing permanently hosted copies of policy PDFs/files.
• https://s3.console.aws.amazon.com/s3/buckets/covid-npi-policy-storage?prefix=Distancing-Status/. Folder in COVID AMP storage S3 bucket containing static CSVs of COVID AMP distancing levels. This is no longer updated.
• https://github.com/CSSEGISandData/COVID-19. GitHub of Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE) which is data source for country-level COVID-19 cases/deaths.
• https://github.com/nytimes/covid-19-data. GitHub of New York Times (NYT) which is data source for state- and county-level COVID-19 cases/deaths.

Code organization

A description of the most important modules and packages in covid-npi-policy-api follows.

Key modules and packages

• amp.py. The main entrypoint for the command line interface (CLI) for data management. Try python -m amp --help from your virtual environment to see instructions, or check the COVID AMP data update brief checklist.
• api. Package containing main API functionality, including defining the routing, API documentation, and functions that retrieve data from the database and return it as API responses.
• db and db_metric. Packages that handle getting a connection to the main COVID AMP database (containing policy data) and the COVID AMP metrics database (containing COVID-19 caseload/death data). Each contains a models.py module that defines the entities and data fields in the databases.
• ingest. Data ingest code, including packages for ingesting data from certain sources, processing data, tagging data, and more. Most data management can be done through the CLI and you needn't run any code in this package, normally.
• main.py. The main entrypoint module of the application (see checklist below for how to start it).

Additional modules and packages

• cli. Packages that support CLI functionality. You should not need to run anything in this package directly as it is called through the CLI.
• dbdocs. Work in progress interactive database documentation created with SchemaSpy. Open the package directory for more information.
• reqs. For those wishing to create a virtual environment with venv instead of pipenv, this directory contains a requirements.txt listing all dev dependencies.

Common tasks for extending and maintaining code

See Common tasks for extending and maintaining code

Checklist to start API server

A list of what is needed and a checklist of steps to start the COVID AMP API server follows.

What you need for this checklist

• An Airtable API key (if you plan to perform data updates) and access to the Airtable base Covid Policy Tracker (see Hailey). If you already have access to this base, your API key can be found on your Airtable account page.

Checklist

1. Install pipenv if you have not already (see https://pipenv.pypa.io/en/latest/install/). Alternatively, you can use Python's to set up your virtual environment and install dev dependencies with pip install -r reqs/requirements.txt (not currently recommended).
2. Install Python v3.7.13 if you have not already; pyenv is a tool for managing multiple Python versions on a single host (see https://github.com/pyenv/pyenv)
3. Install dev packages by doing

   pipenv install --dev --python=3.7.13
   

4. Start a shell in pipenv with your virtual environment with command

   pipenv shell
   

5. If you'd like to connect to a local version of the AMP database, clone the production databases with the following commands, replacing [YOUR_POSTGRESQL_USERNAME] with your local PostgreSQL server username. Otherwise, continue (not recommended).

   python -m amp database clone-from-cloud -u [YOUR_LOCAL_POSTGRESQL_USERNAME] -d covid-npi-policy-local -dc covid-npi-policy
   

   python -m amp database clone-from-cloud -u [YOUR_LOCAL_POSTGRESQL_USERNAME] -d metric-amp-local -dc metric-amp
   

6. Set up environment configuration as described in the next section
7. Start API server with terminal command uvicorn main:app --reload --port 8002. You can then view interactive documentation for the API server at http://localhost:8002/docs.The frontend will make requests to http://localhost:8002, and the API server will restart if you save any Python files it uses.

Environment configuration

Environment variables usable in the application are defined below.

Database connection environment variables

You must at a minimum define environment variable database with the database name for the API server to connect to. Use the name covid-npi-policy-test for the cloud development database or covid-npi-policy for the cloud production database. However, it is recommended you clone one of these to work with locally to avoid making unintended changes to the cloud databases.

By default the API server will use your AWS credentials to access the production database and serve those data. If you'd like to change which host/database/etc. the server connects to, set the following environment variables as needed. If you're using pipenv you can put them in a file .env in the repository root (which git won't track).

host=localhost
port=5432
database=covid-npi-policy-local
database_metric=metric-amp-local
username=[YOUR_LOCAL_POSTGRESQL_USERNAME]
password=[YOUR_LOCAL_POSTGRESQL_PASSWORD]

Note that you must also set environment variable AIRTABLE_API_KEY if you're doing data ingest.

Checklist to perform data updates

See COVID AMP data update brief checklist