Draft to extend metrics with daily/monthly/yearly stats (#3)

* First draft to extend metrics with daily/monthly/yearly stats * Use TypeScript * Update `series` and `ranking` functions * Add types * Remove unused tests * Update README.md * Update dashboard * Update README.md * Use sls-analytics name again
sbstjn · Aug 26, 2017 · e057f44 · e057f44
1 parent 1e8f9f4
commit e057f44
Show file tree

Hide file tree

Showing 17 changed files with 1,505 additions and 171 deletions.
diff --git a/.gitignore b/.gitignore
@@ -4,4 +4,7 @@ jspm_packages
 
 # Serverless directories
 .serverless
+build
 dist
+
+*.log
diff --git a/README.md b/README.md
@@ -5,25 +5,33 @@
 
 Example project for a personal [serverless](https://serverless.com) *Google Analytics* clone to track website visitors.
 
-After deploying the service you will have a HTTP endpoint using API Gateway that accepts requests and puts them into a Kinesis Stream. A Lambda function processes the stream and writes a basic metric about how many visitors you have per absolute URL to DynamoDB.
+After deploying the service you will have an HTTP endpoint using Amazon API Gateway that accepts requests and puts them into a Kinesis Stream. A Lambda function processes the stream and writes basic metrics about how many visitors you have per absolute URL to DynamoDB. 
+
+To access the tracked data, a basic dashboard with a JSON API is included as well. This should be a perfect starting point for you to create your own analytics service.
 
 ## Components
 
 #### Tracking Service
 
 - Amazon Kinesis to stream visitor events
-- Amazon API Gateway as HTTP proxy for Kinesis
+- Amazon API Gateway for Kinesis HTTP proxy
 - Amazon DynamoDB for data storage
 - AWS Lambda to process visitor events
 
-#### Examples
+#### Dashboard
+
+- Amazon API Gateway for JSON API
+- [Dashboard w/ API to show metrics](http://sls-analytics-website-dashboard.s3-website-us-east-1.amazonaws.com/)
+
+#### Example
 
 - [Static website to track visitors](http://sls-analytics-website-example.s3-website-us-east-1.amazonaws.com)
-- [API & Dashboard to show metrics](http://sls-analytics-website-dashboard.s3-website-us-east-1.amazonaws.com/)
+
+*The use of two API Gateways (data tracking and reading) is intended. You might have different settings for tracking and data access when you build something meaningful out of this example.*
 
 ## Configuration
 
-All settings can be changes in the `serverless.yml` configuration file. You can easily change the DynamoDB Table, Kinesis Stream and API Gateway Resource names:
+All settings can be customized in the `serverless.yml` configuration file. You can easily change the DynamoDB Table, Kinesis Stream and API Gateway tracking resource name:
 
 ```yaml
 service: sls-analytics
@@ -38,13 +46,13 @@ custom:
     kinesis: ${self:service}-stream
 ```
 
-The S3 Bucket configuration is only needed for the included example website. If you don't need the example sites, have a look at the `scripts/deploy.sh` file and disable the deployment and remove the CloudFormation resources from the `serverless.yml` file.
+The S3 Bucket configuration is only needed for the included example website and dashboard. If you don't need the examples, have a look at the `scripts/deploy.sh` file and disable the deployment and remove the CloudFormation resources from the `serverless.yml` file.
 
 *Amazon requires unique names for S3 buckets and other resources. Please rename at least the service before you try to deploy the example!*
 
 ## Deployment
 
-Running `yarn deploy` will trigger a default [serverless](https://serverless.com) deployment. After the output of the CloudFormation Stack is available, the included static websites will be generated *(Using the hostname from the stack output)* and uploaded to the configured S3 buckets. As the final step, the deploy process will display the URL of the example website and data dashboard:
+Running `yarn deploy` will trigger a [serverless](https://serverless.com) deployment. After the output of your CloudFormation Stack is available, the included static websites will be generated *(using the hostname from the stack output)* and uploaded to the configured S3 buckets. As the last step, the deploy process will display the URLs of the example website and dashboard:
 
 ```bash
 # Install dependencies
@@ -58,46 +66,93 @@ Dashboard:  http://sls-analytics-website-dashboard.s3-website-us-east-1.amazonaw
 Website:    http://sls-analytics-website-example.s3-website-us-east-1.amazonaws.com/
 ```
 
-The **website** includes a simple HTML file, some stylings, and a few JavaScript lines that send a request to your API on every page load. Visit your URL, hit a few times the refresh button and take a look at the DynamoDB table or the **dashboard** URL.
+The **website** includes a simple HTML file, some stylings, and a few JavaScript lines that send a request to the tracking API on every page load. Open the URL in a web browser, hit a few times the refresh button and take a look at the DynamoDB table or the **dashboard** URL.
 
 ## Tracking
 
-Basically tracking is nothing more than a HTTP request to the API Gateway with a set of payload information *(currently just `url` and `name`)*. Normally you would have a non-JS fallback, like an image e.g., but a simple `fetch` call does the job for now:
+Basically, tracking is nothing more than sending a HTTP request to the API with a set of payload information *(currently `url`, `date`, `name`, and a `websiteId`)*. Normally you would have an additional non-JS fallback, like an image e.g., but a simple `fetch` call does the job for now:
 
 ```js
 fetch(
   'https://n6q0egpreh.execute-api.us-east-1.amazonaws.com/v1/track',
   {
     method: "POST",
-    body: JSON.stringify( { url: location.href, name: document.title } ),
-    headers: new Headers(
+    body: JSON.stringify(
       {
-        "Content-Type": "application/json"
+        date: new Date().getTime(),
+        name: document.title,
+        url: location.href,
+        website: 'yfFbTv1GslRcIkUsWpa7'
       }
-    )
+    ),
+    headers: new Headers({ "Content-Type": "application/json" })
   }
 )
 ```
 
 ## Data Access
 
-You can get a list of all tracked data by invoking the `list` function, or just have a look into your DynamoDB or use the included dashboard.
+An example [dashboard to access tracked data](http://sls-analytics-website-dashboard.s3-website-us-east-1.amazonaws.com/) is included and deployed to S3. The URL will be displayed after the `deploy` task. You can access the metrics using basic `curl` requests as well. Just provide the `website` and `date` parameters:
+
+### Top Content
+
+The `ranking` resource scans the DynamoDB for pages with the most hits on a specific `date` value:
 
 ```bash
-$ > sls invoke -f list
+$ > curl https://p026537a2j.execute-api.us-east-1.amazonaws.com/dev/ranking?website=yfFbTv1GslRcIkUsWpa7&date=MONTH:2017-08
 
 [
   {
-    "name": "http://sls-analytics-website-example.s3-website-us-east-1.amazonaws.com/",
+    "name": "Example Website - Serverless Analytics",
+    "url": "http://sls-analytics-website.s3-website-us-east-1.amazonaws.com/baz",
+    "value": 19
+  },
+  {
+    "name": "Example Website - Serverless Analytics",
+    "url": "http://sls-analytics-website.s3-website-us-east-1.amazonaws.com/",
     "value": 10
   },
   {
-    "name": "http://sls-analytics-website-example.s3-website-us-east-1.amazonaws.com/?foo=bar",
-    "value": 2
+    "name": "Example Website - Serverless Analytics",
+    "url": "http://sls-analytics-website.s3-website-us-east-1.amazonaws.com/bar",
+    "value": 4
   }
 ]
 ```
 
+### Requests per URL
+
+The `series` resource scans the DynamoDB for data about a specific `url` in a given `date` period.
+
+```bash
+$ > curl https://p026537a2j.execute-api.us-east-1.amazonaws.com/dev/series?website=yfFbTv1GslRcIkUsWpa7&date=HOUR:2017-08-25T13&url=http://sls-analytics-website.s3-website-us-east-1.amazonaws.com/baz
+
+[
+  {
+    "date": "MINUTE:2017-08-25T13:33",
+    "value": 1
+  },
+  {
+    "date": "MINUTE:2017-08-25T13:37",
+    "value": 1
+  },
+  {
+    "date": "MINUTE:2017-08-25T13:46",
+    "value": 14
+  },
+  {
+    "date": "MINUTE:2017-08-25T13:52",
+    "value": 1
+  }
+]
+```
+
+### Date Parameter
+
+The DynamoDB stores the absolute hits number for the dimensions `YEAR`, `MONTH`, `DATE`, `HOUR`, and `MINUTE` per default. This may cause lots of write capacities when processing events, but with the [serverless-dynamodb-autoscaling](https://github.com/sbstjn/serverless-dynamodb-autoscaling) plugin DynamoDB will scale the capacities when needed.
+
+All dates are UTC values!
+
 ## Infrastructure
 
 ![Infrastructure](infra.png)

diff --git a/package.json b/package.json
@@ -2,14 +2,33 @@
   "license": "MIT",
   "scripts": {
     "build": "./scripts/build.sh",
-    "deploy": "./scripts/deploy.sh"
+    "deploy": "./scripts/deploy.sh",
+    "lint": "tslint src/**/*.ts",
+    "test": "jest test",
+    "test:watch": "jest test --watch"
+  },
+  "jest": {
+    "transform": {
+      ".*": "<rootDir>/node_modules/ts-jest/preprocessor.js"
+    },
+    "testRegex": "(/__tests__/.*|\\.(test|spec))\\.(ts)$",
+    "moduleFileExtensions": [
+      "ts",
+      "js"
+    ]
   },
   "devDependencies": {
+    "@types/node": "^8.0.25",
+    "aws-sdk": "^2.104.0",
     "ejs-cli": "^2.0.0",
+    "jasmine-data-provider": "^2.2.0",
     "node-sass": "^4.5.3",
     "s3sync-cli": "^2.0.0",
-    "serverless": "^1.17.0",
+    "serverless": "^1.20.2",
     "serverless-dynamodb-autoscaling": "^0.6.1",
-    "serverless-stack-output": "0.2"
+    "serverless-plugin-typescript": "^1.1.1",
+    "serverless-stack-output": "^0.2.0",
+    "ts-jest": "^20.0.10",
+    "typescript": "^2.4.2"
   }
-}
+}
diff --git a/serverless.yml b/serverless.yml
@@ -1,8 +1,9 @@
 service: sls-analytics
 
 plugins:
-  - serverless-stack-output
   - serverless-dynamodb-autoscaling
+  - serverless-plugin-typescript
+  - serverless-stack-output
 
 provider:
   name: aws
@@ -19,10 +20,11 @@ provider:
 
 custom:
   stage: ${opt:stage, self:provider.stage}
+  website: yfFbTv1GslRcIkUsWpa7 # Random ID
   names:
     bucket: 
-      website: ${self:service}-website-example
-      dashboard: ${self:service}-website-dashboard
+      website: ${self:service}-website
+      dashboard: ${self:service}-dashboard
     resource: track
     dynamodb: ${self:service}-data
     kinesis: ${self:service}-stream
@@ -31,11 +33,11 @@ custom:
   capacities:
     - table: DynamoDBTable
       read:
-        minimum: 5
-        maximum: 100
+        minimum: 10
+        maximum: 120
       write:
-        minimum: 5
-        maximum: 100
+        minimum: 10
+        maximum: 120
 
 package:
   exclude:
@@ -52,14 +54,18 @@ package:
     - yarn.lock
 
 functions:
-  list:
-    handler: src/list.get
-
-  route-list:
-    handler: src/route/list.get
+  series:
+    handler: src/api/series.get
+    events:
+      - http:
+          path: /series
+          method: GET
+
+  ranking:
+    handler: src/api/ranking.get
     events:
       - http:
-          path: /list
+          path: /ranking
           method: GET
 
   process:
@@ -97,6 +103,9 @@ resources:
     Region:
       Description: AWS Region
       Value: { Ref: 'AWS::Region' }
+    WebsiteID:
+      Description: Website ID
+      Value: ${self:custom.website}
 
   Resources:
     KinesisStream:
@@ -110,11 +119,15 @@ resources:
       Properties:
         TableName: ${self:custom.names.dynamodb}
         AttributeDefinitions:
-          - AttributeName: url
+          - AttributeName: id
+            AttributeType: S
+          - AttributeName: date
             AttributeType: S
         KeySchema:
-          - AttributeName: url
+          - AttributeName: id
             KeyType: HASH
+          - AttributeName: date
+            KeyType: RANGE
         ProvisionedThroughput:
           ReadCapacityUnits: 5
           WriteCapacityUnits: 5
@@ -144,7 +157,7 @@ resources:
         BucketName: ${self:custom.names.bucket.website}
         WebsiteConfiguration:
           IndexDocument: index.html
-          ErrorDocument: error.html
+          ErrorDocument: index.html
 
     BucketWebsitePolicy:
       Type: AWS::S3::BucketPolicy

diff --git a/sites/dashboard/index.ejs b/sites/dashboard/index.ejs
@@ -7,6 +7,7 @@
     <link href="/style.css" media="all" rel="stylesheet" />
   </head>
   <body>
+    <div class="loader">Loading...</div>
     <div id="container">
       <table>
         <thead>
@@ -38,12 +39,13 @@
         item => '<tr><td><strong>' + item.name + '</strong><br />' + item.url + '</td><td>' + item.value + '</td></tr>'
       )
 
+      document.querySelector('.loader').style.display = 'none'
       document.querySelector('#data').innerHTML = rows.join('')
       document.querySelector('#container').classList.add('show')
     } 
 
     fetch(
-      '<%=APIRootURL%>/<%=APIStageName%>/list'
+      '<%=APIRootURL%>/<%=APIStageName%>/ranking?website=<%=WebsiteID%>&date=YEAR:2017'
     ).then(
       res => res.json()
     ).then(

diff --git a/sites/dashboard/style.scss b/sites/dashboard/style.scss
@@ -104,3 +104,41 @@ footer {
     }
   }
 }
+
+.loader {
+  margin: 100px auto;
+  font-size: 8px;
+  width: 1em;
+  height: 1em;
+  border-radius: 50%;
+  position: relative;
+  text-indent: -9999em;
+  -webkit-animation: load5 1.1s infinite ease;
+  animation: load5 1.1s infinite ease;
+  -webkit-transform: translateZ(0);
+  -ms-transform: translateZ(0);
+  transform: translateZ(0);
+}
+
+@-webkit-keyframes load5 {
+  0%,
+  100% { box-shadow: 0em -2.6em 0em 0em #cccccc, 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.5), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.7); }
+  12.5% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.7), 1.8em -1.8em 0 0em #cccccc, 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.5); }
+  25% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.5), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.7), 2.5em 0em 0 0em #cccccc, 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  37.5% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.5), 2.5em 0em 0 0em rgba(200, 200, 200, 0.7), 1.75em 1.75em 0 0em #cccccc, 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  50% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.5), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.7), 0em 2.5em 0 0em #cccccc, -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  62.5% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.5), 0em 2.5em 0 0em rgba(200, 200, 200, 0.7), -1.8em 1.8em 0 0em #cccccc, -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  75% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.5), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.7), -2.6em 0em 0 0em #cccccc, -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  87.5% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.5), -2.6em 0em 0 0em rgba(200, 200, 200, 0.7), -1.8em -1.8em 0 0em #cccccc; }
+}
+@keyframes load5 {
+  0%,
+  100% { box-shadow: 0em -2.6em 0em 0em #cccccc, 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.5), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.7); }
+  12.5% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.7), 1.8em -1.8em 0 0em #cccccc, 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.5); }
+  25% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.5), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.7), 2.5em 0em 0 0em #cccccc, 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  37.5% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.5), 2.5em 0em 0 0em rgba(200, 200, 200, 0.7), 1.75em 1.75em 0 0em #cccccc, 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  50% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.5), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.7), 0em 2.5em 0 0em #cccccc, -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.2), -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  62.5% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.5), 0em 2.5em 0 0em rgba(200, 200, 200, 0.7), -1.8em 1.8em 0 0em #cccccc, -2.6em 0em 0 0em rgba(200, 200, 200, 0.2), -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  75% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.5), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.7), -2.6em 0em 0 0em #cccccc, -1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2); }
+  87.5% { box-shadow: 0em -2.6em 0em 0em rgba(200, 200, 200, 0.2), 1.8em -1.8em 0 0em rgba(200, 200, 200, 0.2), 2.5em 0em 0 0em rgba(200, 200, 200, 0.2), 1.75em 1.75em 0 0em rgba(200, 200, 200, 0.2), 0em 2.5em 0 0em rgba(200, 200, 200, 0.2), -1.8em 1.8em 0 0em rgba(200, 200, 200, 0.5), -2.6em 0em 0 0em rgba(200, 200, 200, 0.7), -1.8em -1.8em 0 0em #cccccc; }
+}
diff --git a/sites/website/index.ejs b/sites/website/index.ejs
@@ -31,7 +31,14 @@
       '<%=TrackingRootURL%>/<%=TrackingStageName%>/<%=TrackingResource%>',
       {
         method: "POST",
-        body: JSON.stringify( { url: location.href, name: document.title } ),
+        body: JSON.stringify(
+          {
+            date: new Date().getTime(),
+            name: document.title,
+            url: location.href,
+            website: '<%=WebsiteID%>'
+          }
+        ),
         headers: new Headers(
           {
             "Content-Type": "application/json"
-Original file line number
+Diff line change
@@ Expand Up / @@ -4,4 +4,7 @@ jspm_packages @@
     # Serverless directories
     .serverless
+    build
     dist
+    *.log