Sync to latest covid green (#2)

* Revert to original interop lambdas for compatability, support upload to and download from multiple backends (#25) * Use midnight UTC as cut-off for exposure key expiration (#24) * Set missing columns when re-inserting invalid keys (#26) * feat: support for notices in cleanup lambda (#27) * Use correct timezone for inserted metrics (covidgreen#28) * close db connections to avoid flooding db (covidgreen#29) * Syncing to latest (#6) * Support for APHL verification process (#10) * Applies linting rules in line with other covid-green repos * Enforces linting on all PRs and pushes to current * Don't include keys which are still valid, re-insert them for future processing (#13) * Fix error thrown when processing an invalid key from interop (#20) * Cleanup lambda (#19) * EFGS compatability (#22) * Remove exp for tokens as they should be manually revoked in db (#21) * Add offset to check for still valid keys, option to disable check (#23) * Revert to original interop lambdas for compatability, support upload to and download from multiple backends (#25) * Use midnight UTC as cut-off for exposure key expiration (#24) * Set missing columns when re-inserting invalid keys (#26) * feat: support for notices in cleanup lambda (#27) * Use correct timezone for inserted metrics (covidgreen#28) * close db connections to avoid flooding db (covidgreen#29) Co-authored-by: Jack Murdoch <[email protected]> Co-authored-by: Dennis Gove <[email protected]> Co-authored-by: Paolo Chiodi <[email protected]> * latest (#7) * Support for APHL verification process (#10) * Applies linting rules in line with other covid-green repos * Enforces linting on all PRs and pushes to current * Don't include keys which are still valid, re-insert them for future processing (#13) * Fix error thrown when processing an invalid key from interop (#20) * Cleanup lambda (#19) * EFGS compatability (#22) * Remove exp for tokens as they should be manually revoked in db (#21) * Add offset to check for still valid keys, option to disable check (#23) * Revert to original interop lambdas for compatability, support upload to and download from multiple backends (#25) * Use midnight UTC as cut-off for exposure key expiration (#24) * Set missing columns when re-inserting invalid keys (#26) * feat: support for notices in cleanup lambda (#27) * Use correct timezone for inserted metrics (covidgreen#28) * close db connections to avoid flooding db (covidgreen#29) Co-authored-by: Jack Murdoch <[email protected]> Co-authored-by: Dennis Gove <[email protected]> Co-authored-by: Paolo Chiodi <[email protected]> * proto update * Latest changes * Syncing to latest changes (covidgreen#30) * Merge changes from Nearform repos * Deploy interop lambdas to cti-dev * Improve logging, rollback batch on upload error (#2) * Fix transaction * Deployment actions for qa-dela * Make token description mandatory * Create metrics for number of keys uploaded and downloaded to and from the interop service * Upgrade pg to latest * Fix start and end timestamps in export files * Deployment actions for pa-prod (#4) * Deployment actions for prod-scot (#5) * Deployment actions for prod-jers (#6) * Deployment actions for prod-dela (#7) * Remove deployment actions for dev-ni * Remove dev-gi deployment actions * Deployment actions for NJ QA (#8) * Deployment actions for cleanup lambda * Deployment actions for nfcs-dev (#9) * nfcs-dev does not have interop lambdas * Missing eployment actions for cleanup lambda * Deployment actions for prod-newjers (#10) * Revert to original interop lambdas to prevent problems in prod * add annotations to dev * Update prod.yml * add credentials * add credentials * Deploy interop integration on QA envs * Deployment actions for qa-gct (#14) * Deploy interop lambdas to gct-qa * Deploy interop lambdas to gct-qa * Temporary EFGS integration (#15) * Remove dev annotation * Stop prod-cti deploys temporarily * Deploy interop lambdas to new envs * Better handling for no new keys * Upload to interop in batches of 1000 (#16) * Handle missing transmissionRiskLevel in EFGS * Deploy interop to gct-prod * Download new batches from EFGS only (#17) * feat: support for notices in cleanup lambda (#18) * Enable cti-prod deployment * QR code generation (#20) * Fix var name for qr bucket * Export file version (#23) * Ensure daysSinceOnset and transmissionRiskLevel are valid (#24) * Exclude temporary records * Support for alerting when upload threshold exceeded (#25) * Missing await * Missing imports * Logging * Await db * Fix missing reference * More logging, fix query * Fix query and limit to check-ins at current venue * Fix query * Alert date formatting improvement * fix: Remove erroneous code * Exposure fix * Lint fix * More exposure changes * fixing exposure check * Fix for daily file generation * Fix for daily file generation * Reverting exposure change and adding cf lambda * reverting exposure changes * Switching exposures to daily history file * Lint fiox * Exposures fix * Fixing order of file creation * Fixing daily files * removing unused lambda * Fixing origin * lint fixes * Removing log * Default fix * Adding check for future keys * adding download count fix * lint fix * Adding package lock * Fixing variance * Lint fix * Fixing upload check * Fixing data param * Fixing key name * Removing unused code * adding enx logo metrcis generation * lint fix * Minro tweaks * Only run enx logo counts if enabled Co-authored-by: Jack Murdoch <[email protected]> Co-authored-by: Jack Murdoch <[email protected]> Co-authored-by: Valerio Barrila <[email protected]> Co-authored-by: Paolo Chiodi <[email protected]> * Update cleanup.js Co-authored-by: Jack Murdoch <[email protected]> Co-authored-by: Paolo Chiodi <[email protected]> Co-authored-by: Dennis Gove <[email protected]> Co-authored-by: Jack Murdoch <[email protected]> Co-authored-by: Valerio Barrila <[email protected]>
NES-Digital-Service · Feb 10, 2021 · 361e283 · 361e283
1 parent 987b8d2
commit 361e283
Show file tree

Hide file tree

Showing 14 changed files with 1,056 additions and 388 deletions.
diff --git a/.env.sample b/.env.sample
@@ -19,7 +19,17 @@ INTEROP_URL=
 INTEROP_TOKEN=
 INTEROP_MAX_AGE=
 INTEROP_PRIVATE_KEY=
-INTEROP_CERTIFICATE=
+INTEROP_SERVER_ID=
+
+INTEROP_ORIGIN=
+
+EFGS_URL=
+EFGS_AUTH_CERT=
+EFGS_AUTH_KEY=
+EFGS_SIGN_CERT=
+EFGS_SIGN_KEY=
+EFGS_DOWNLOAD=true
+EFGS_UPLOAD=true
 
 APP_BUNDLE_ID=
 APP_PACKAGE_NAME=
@@ -34,3 +44,14 @@ VARIANCE_OFFSET_MINS=120
 
 CODE_LIFETIME_MINS=10
 UPLOAD_TOKEN_LIFETIME_MINS=1440
+NOTICE_LIFETIME_MINS=20160
+
+QR_SENDER=
+QR_BUCKET_NAME=
+QR_APP_URL=
+QR_ALERT_EMAIL=
+
+TIME_ZONE=
+
+ENX_LOGO_SUPPORTED=
+
diff --git a/.gitignore b/.gitignore
@@ -18,3 +18,5 @@ yarn.lock
 .env
 
 lambdas.zip
+out
+pdf-assets/*.pdf
diff --git a/README.md b/README.md
@@ -18,6 +18,7 @@ There are a number of handy commands you can run to help with development.
 | `npm run test:token:metrics` | Run test for token specific to metrics |
 | `npm run test:token:push` | Run test for token specific to push |
 | `npm run test:token:register` | Run test for token specific to register |
+| `npm run test:cleanup` | Run cleanup for local env |
 | `npm run lint` | Run eslint |
 | `npm run lint:fix` | Run eslint in fix mode |
 

diff --git a/authorizer.js b/authorizer.js
@@ -2,6 +2,7 @@ const { isAuthorized, runIfDev } = require('./utils')
 
 exports.handler = async function(event) {
   const secret = process.env.JWT_SECRET
+
   if (!secret) {
     console.log('Error acquiring secret from env')
 

diff --git a/cleanup.js b/cleanup.js
@@ -1,22 +1,152 @@
 const SQL = require('@nearform/sql')
-const { getDatabase, getExpiryConfig, runIfDev } = require('./utils')
+const AWS = require('aws-sdk')
+const { utcToZonedTime } = require('date-fns-tz')
+
+const {
+  withDatabase,
+  getExpiryConfig,
+  getTimeZone,
+  getENXLogoEnabled,
+  runIfDev
+} = require('./utils')
 
 async function createRegistrationMetrics(client) {
+  const timeZone = await getTimeZone()
+
   const sql = SQL`
     INSERT INTO metrics (date, event, os, version, value)
-    SELECT CURRENT_DATE, 'REGISTER', '', '', COUNT(id)
-    FROM registrations WHERE created_at::DATE = CURRENT_DATE
+    SELECT
+      (CURRENT_TIMESTAMP AT TIME ZONE ${timeZone})::DATE,
+      'REGISTER',
+      '',
+      '',
+      COUNT(id)
+    FROM registrations
+    WHERE
+      (nonce != '123456' OR nonce IS NULL)
+    AND
+      (created_at AT TIME ZONE ${timeZone})::DATE =
+      (CURRENT_TIMESTAMP AT TIME ZONE ${timeZone})::DATE
     ON CONFLICT ON CONSTRAINT metrics_pkey
     DO UPDATE SET value = EXCLUDED.value
-    RETURNING value
-  `
+    RETURNING value`
 
   const { rows } = await client.query(sql)
   const [{ value }] = rows
 
   console.log(`updated register metric for today with value ${value}`)
 }
 
+async function storeENXLogoRequests(client, metrics) {
+  const timeZone = await getTimeZone()
+
+  // include zero metrics also for now
+  const nonZeroMetrics = metrics // .filter(m => m.value > 0)
+
+  const sql = SQL`
+    INSERT INTO metrics (date, event, os, version, value)
+    VALUES `
+
+  nonZeroMetrics.forEach((metric, index) => {
+    sql.append(
+      SQL`((CURRENT_TIMESTAMP AT TIME ZONE ${timeZone})::DATE, ${metric.metric}, '', '', ${metric.value})`
+    )
+    if (index < nonZeroMetrics.length - 1) {
+      sql.append(SQL`,`)
+    }
+  })
+
+  sql.append(SQL`
+    ON CONFLICT ON CONSTRAINT metrics_pkey
+    DO UPDATE SET value = EXCLUDED.value
+    WHERE metrics.date = EXCLUDED.date AND metrics.event = EXCLUDED.event
+  `)
+
+  if (nonZeroMetrics.length > 0) {
+    await client.query(sql)
+  }
+}
+
+function buildMetricsQuery() {
+  const metrics = [
+    { metric: 'enxlogoall', label: 'ENX_LOGO_REQUESTS_ALL' },
+    { metric: 'enxlogo200', label: 'ENX_LOGO_REQUESTS_200' },
+    { metric: 'enxlogo304', label: 'ENX_LOGO_REQUESTS_304' },
+    { metric: 'enxlogosettings', label: 'ENX_LOGO_REQUESTS_SETTINGS' },
+    { metric: 'enxlogoenbuddy', label: 'ENX_LOGO_REQUESTS_ENBUDDY' }
+  ]
+  const metricsData = []
+
+  metrics.forEach(m => {
+    metricsData.push({
+      Id: `en_${m.metric}`,
+      MetricStat: {
+        Metric: {
+          Namespace: 'ApiGateway',
+          MetricName: m.metric
+        },
+        Period: 86400,
+        Stat: 'Sum'
+      },
+      Label: `${m.label}`,
+      ReturnData: true
+    })
+  })
+
+  return metricsData
+}
+
+async function createENXLogoMetrics(client, event) {
+  const timeZone = await getTimeZone()
+  const enxLogoEnabled = await getENXLogoEnabled()
+
+  if (!enxLogoEnabled) {
+    console.log('Skipping enx logo checks, not enabled')
+    return
+  }
+
+  const cw = new AWS.CloudWatch()
+
+  let startDate = new Date()
+  startDate.setHours(0, 0, 0, 0)
+
+  if (event && event.startDate) {
+    startDate = new Date(event.startDate)
+  }
+  const endDate = new Date(startDate)
+  endDate.setHours(0, 0, 0, 0)
+  endDate.setDate(endDate.getDate() + 1)
+
+  const params = {
+    MetricDataQueries: buildMetricsQuery(),
+    StartTime: utcToZonedTime(startDate, timeZone),
+    EndTime: utcToZonedTime(endDate, timeZone)
+  }
+  const logData = await new Promise((resolve, reject) => {
+    cw.getMetricData(params, function(err, data) {
+      if (err) {
+        console.log(err) // an error occurred
+        reject(err)
+      } else {
+        resolve(data)
+      }
+    })
+  })
+
+  const results = logData.MetricDataResults
+  const dbMetrics = []
+
+  results.forEach(response => {
+    dbMetrics.push({
+      metric: response.Label,
+      value:
+        response.Values && response.Values.length > 0 ? response.Values[0] : 0
+    })
+  })
+  await storeENXLogoRequests(client, dbMetrics)
+  console.log('updated enx logo requests metrics', startDate, dbMetrics)
+}
+
 async function removeExpiredCodes(client, codeLifetime) {
   const sql = SQL`
     DELETE FROM verifications
@@ -39,13 +169,33 @@ async function removeExpiredTokens(client, tokenLifetime) {
   console.log(`deleted ${rowCount} tokens older than ${tokenLifetime} minutes`)
 }
 
-exports.handler = async function() {
-  const client = await getDatabase()
-  const { codeLifetime, tokenLifetime } = await getExpiryConfig()
+async function removeOldNoticesKeys(client, noticeLifetime) {
+  const sql = SQL`
+    DELETE FROM notices
+    WHERE created_at < CURRENT_TIMESTAMP - ${`${noticeLifetime} mins`}::INTERVAL
+  `
+
+  const { rowCount } = await client.query(sql)
+
+  console.log(
+    `deleted ${rowCount} notices keys older than ${noticeLifetime} minutes`
+  )
+}
+
+exports.handler = async function(event) {
+  const {
+    codeLifetime,
+    tokenLifetime,
+    noticeLifetime
+  } = await getExpiryConfig()
 
-  await createRegistrationMetrics(client)
-  await removeExpiredCodes(client, codeLifetime)
-  await removeExpiredTokens(client, tokenLifetime)
+  await withDatabase(async client => {
+    await createRegistrationMetrics(client)
+    await removeExpiredCodes(client, codeLifetime)
+    await removeExpiredTokens(client, tokenLifetime)
+    await removeOldNoticesKeys(client, noticeLifetime)
+    await createENXLogoMetrics(client, event)
+  })
 
   return true
 }
-Original file line number
+Diff line change
@@ Expand Up / @@ -18,3 +18,5 @@ yarn.lock @@
     .env
     lambdas.zip
+    out
+    pdf-assets/*.pdf