PDF Inventory where title is PMID

[joshlawrimore]

Josh will upload the existing inventory of PDFs to the S3 bucket osm-pdf-uploads. Each PDF from the IRP inventory will have PMID as its filename. Task is to:

Write code that parses the S3 bucket’s inventory to create a document entry, a works entry, and a identifier entry based on the document filehash and the PMID in the title. The upload should create an entry in the provenance table and the provenance_id should be included in the works, document, and identifier tables.

Code should be written to iterate over S3 inventory to ensure that PDF inventory in S3 and document’s table are synced. Any missing file hashes should be able to be reported and either added or removed from the documents table. If a PDF is removed, then all references to a missing pdf are removed from foreign keys from tables.
Run the oddpub code on the inventory to generate the oddpub metrics table.

The resultant database should be in an AWS postgres database.

[joshlawrimore]

@quang-ng and @leej3 , I will start with 3539 PDFs that I have on my local machine. I will then write code to pull the existing inventory from biowulf to the osm-pd-uploads bucket

[joshlawrimore]

@quang-ng and @leej3, the 3538 pdfs are in the osm-pdf-uploads bucket in the pdfs subdirectory.