Google Analytics anomalies monitor

Application allows detection of anomalies in Google Analytics data based on custom events. Forecast data is calculated using Holt's linear trend method (exponential smoothing). Here is code base which provides API to calculate and manage data. Check out client application which is UI for Google Analytics anomalies monitor.

Build

Dependencies:

• nodejs,
• npm.

Clone code and install app dependencies:

git clone https://github.com/fraszczakszymon/ga-anomalies-monitor.git
cd ga-anomalies-monitor
npm install --production

Create configuration files:

Server configuration:

Copy file with basic config:

cp config/config.sample.json config/config.json

Port provided in this file will be used to run API server - make sure it's not in use.

Google Analytics credentials:

cp config/credentials.sample.json config/credentials.json

Steps to create your credentials:

1. Go to Google Developers Console,

2. Create a new project,

3. Enable Analytics API,

4. Go to management of service accounts and create a new service account with a private key (type: p12),

5. Save automatically downloaded .p12 file in config/key.p12,

6. Create config/key.pem file using command:

    cat config/key.p12 | openssl pkcs12 -nodes -nocerts -passin pass:notasecret | openssl rsa > config/key.pem
   

7. Save newly created e-mail address in accountEmail field of config/credentials.json file. Example below:

    {
      "accountEmail": "[email protected]",
      "accountKey": "config/key.pem"
    }
   

8. Grant permissions to the same e-mail address on Google Analytics. It may have read-only privileges.

Verify credentials:

Run command:

node cli.js profiles

And check whether all available GA profiles are listed.

Run server:

npm start

Queries configuration

Build queries with the syntax described in documentation. Each json with query configuration should be placed in config/config.json in queries array. To verify query you can use GA Query Explorer. Time dimensions (day, hour) are added automatically so correct configuration should look like:

{
    "id": "pageview",
    "title": "Pageviews",
    "description": "Number of pageviews",
    "viewIds": "12345678",
    "metrics": [
        "pageviews"
    ],
    "dimensions": [],
    "filters": "",
    "alpha": 0.9,
    "beta": 0.6,
    "threshold": 0.15,
    "enabled": true
},
{
    "id": "my-custom-event",
    "title": "My custom event",
    "description": "Events with category here/is/category and action here.is.action.name",
    "viewIds": "12345678",
    "metrics": [
        "totalEvents"
    ],
    "dimensions": [
    	"eventCategory"
    ],
    "filters": "ga:eventCategory=@here/is/catego;ga:eventAction==here.is.action.name",
    "alpha": 0.9,
    "beta": 0.8,
    "threshold": 0.2,
    "enabled": true
}

Description of parameters:

Parameter	Type	Description
id	string	Unique query id
title	string	Title to be displayed in UI panel
description	string	Description to be displayed in UI panel
viewIds	integer	GA profile id
metrics, dimensions, filters	string	GA filter parameters
threshold	float	Decide when error should be marked as an anomaly
alpha, beta	float	Used to calculate forecast data. Value has to be in range from 0.01 to 1.0. You can choose these values experimentally or using command: node cli.js parameters <queryId>
enabled	boolean	Decide whether include query in request to API or skip it

API description

History of builds

GET /build

Returns json with created builds (max 48).

[
	{
		"id": 29,
		"date": "2016-02-05T08:35:12+01:00",
		"duration": 14984,
		"status": 0
	},
	{
		"id": 28,
		"date": "2016-02-05T08:34:22+01:00",
		"duration": 14901,
		"status": 0
	},
	{
		"id": 27,
		"date": "2016-02-05T08:30:22+01:00",
		"duration": 16406,
		"status": 0
	}
]

Field	Type	Description
id	integer	Build id
date	string	Date when build was created
duration	integer	Build duration in milliseconds
status	integer	Current status of build: 0 - done, 1 - pending, 2 - failed

Build details

GET /build/{id}

Returns json with build details and calculated data.

{
	"id": 29,
	"date": "2016-02-05T08:35:12+01:00",
	"duration": 14984,
	"status": 0,
	"queries": [
		{
			"id": "pageview",
			"title": "Pageviews",
			"description": "Number of pageviews",
			"data": [
				{
					"date": "2016-02-01T00:00:00-08:00",
					"value": 345248,
					"error": 0,
					"forecast": 345248,
					"max": 397035.2,
					"min": 293460.8,
					"change": 0,
					"exceeded": false
				},
				{
					"date": "2016-02-01T01:00:00-08:00",
					"value": 306954,
					"error": 0,
					"forecast": 306954,
					"max": 352997.1,
					"min": 260910.9,
					"change": -11.091736954305311,
					"exceeded": false
				}
			],
			"errors": 0
		}
	]
}

Field	Type	Description
id	integer	Build id
date	string	Date when build was created
duration	integer	Build duration in milliseconds
status	integer	Current status of build: 0 - done, 1 - pending, 2 - failed
queries	array	Array with defined queries data

Each query object contains:

Field	Type	Description
id	string	Query id (defined in config)
title	string	Query title (defined in config)
description	string	Query description (defined in config)
data	array	Calculated data details
errors	integer	Number of detected anomalies

Each row in data contains:

Field	Type	Description
date	string	Date with time
value	integer	Number of event occurrences at given time
forecast	integer	Expected number of event occurrences at given time
error	float	Difference between expected and real value of events
max	float	Maximum value of difference range (based on configured threshold)
min	float	Minimum value of difference range (based on configured threshold)
change	float	Percentage of change based on previous row
exceeded	boolean	True if value exceeded defined threshold

Create build

POST /build

Starts new build.

{
	"id": 29,
	"date": "2016-02-05T08:35:12+01:00",
	"duration": 0,
	"status": 1
}

Field	Type	Description
id	integer	Build id
date	string	Date when build was created
duration	integer	Build duration in milliseconds
status	integer	Current status of build: 0 - done, 1 - pending, 2 - failed

Contribution

1. Check out build instructions,
2. Install dev dependencies using npm install,
3. Check out .editorconfig file to keep coding with the same styles guidelines,
4. Before commiting run npm run check to check tests results and all files styles with linter.

License

The MIT License

Copyright (c) 2016 Frąszczak Szymon (@fraszczakszymon)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.