Skip to content
This repository has been archived by the owner on Dec 29, 2019. It is now read-only.

Design

Jump to bottom
Sun Liwen edited this page Sep 1, 2013 · 13 revisions

Tech Stack

  • Server Side
    • Python
    • Django
    • Nginx
    • MongoDB
    • Hadoop/Hive
  • Browser Side
    • Javascript
    • jQuery
    • Highchart
  • OS
    • Ubuntu/Debian
    • Mac OS X

Components

The whole application includes several components.

  • Web Services
    • api service - The core service of the application, it's the inbound and outbound endpoint of all the data collected from consumer websites or generated by the application.
    • admin service - The administration web application to create user, api keys, trigger data calculation, etc.
    • dashboard - The visualization and statistics web application, it visualizes the internal state of the data in an easy way and it's print friendly.
  • System Services
    • batch service - The core calculation service, this service will do all the calculation and analysis on the inbound data using python and hive.
    • hadoop - The layer below batch service, which provides the ability to do map/reduce, good for manipulate massive of dateset.

Deployments

Web Services

All the web services can be deployed on a single server to reduce cost, and some inhouse deployment has kept running for 2 years without big problem.

System Services

Batch is a computing centric service which should be better deployed on a powerful machine with strong CPUs and huge memory. Powerful computing resource will reduce the computing time effectively.

Hadoop is an underlying service which can be build as a cluster or standalone which will be used by batch service internally.

Features Introduction

APIs

The core APIs involved in collecting data points from consumer websites and providing consumable data.

There are about 20 apis and counting:

  • viewItem
  • addFavorite
  • removeFavorite
  • rateItem
  • removeOrderItem
  • placeOrder
  • updateItem
  • removeItem
  • getAlsoViewed
  • getAlsoBought
  • getBoughtTogether
  • getUltimatelyBought
  • getByBrowsingHistory
  • getByPurchasingHistory
  • getByShoppingCart
  • updateCategory
  • getByEachBrowsedItem
  • getByEachPurchasedItem
  • unlike

etc.

Everyone of these APIs can be invoked separately. They also can be packed into one request in order to save server hits and this is the default behavior.

Some request samples:

  • An real request to ask for a product page
var is_on_sale = true;function tjbSyncPage(){
    var p = new tui.Point("API_KEY");                          
    p.addSharedParams('item_id', 'U290001400');                // 商品id, String类型
    p.addSharedParams('user_id', 'null');                      // 用户id, String类型
    p.addSharedParams('amount', 5);

	if(is_on_sale){
        p.addRequest("updateItem",                             // 更新商品
            {
                'item_name': "伊瑞丝--植物配方超温和高级婴儿手口专用湿巾(80片三连包）", // 商品名称, String类型
                'item_link': "http://www.leyou.com.cn/product/single/U290001400",
                                                               // 商品链接, String类型
                'image_link': "http://image.leyou.com.cn/images_db/77/f5/77f51eb59c068b059b112e5e0b106d82.JPG",
                                                               // 商品缩略图链接, String类型
                'market_price': "88.00",                       // 市场价, String类型
                'price': "49.00",                              // 优惠价, String类型, 可选
                'categories': "12,1200,120000"                 // 商品分类, 用半角逗号分割
            }
        );
    } else {
        p.addRequest("removeItem");                            // 停止推荐此商品
    }
    p.addRequest("viewItem");                                  // 记录浏览行为
    p.addRequest("getAlsoBought");                             // 获取还购买过的推荐
    p.addRequest("getAlsoViewed");                             // 获取还浏览过的推荐
    p.addRequest("getUltimatelyBought");                       // 获取最终购买的推荐
    p.invoke("tjbCallback");                                   // 异步发起请求
}

GET /api/1.0/?&api_key=API_KEY&callback=A_CALLBACK&_item_id=U290001400&_user_id=null&_amount=5&-=1600&1378034705184

tjbCallback({
    "code": 0,
    "responses": {
        "getAlsoViewed": {
            "topn": [{
                "item_name": "\u51ef\u745e\u4e1d--80\u7247\u8d85\u6e29\u548c\u578b\u5a74\u513f\u67d4\u6e7f\u5dfe(\u4e09\u8054\u5305)",
                "price": "29.90",
                "market_price": "48.00",
                "image_link": "http://image.leyou.com.cn/images_db/b6/a9/b6a92fae73e1f6e63ce0890146cb6e4d.JPG",
                "score": 0.99768716940000002,
                "item_link": "http://api.leyou.com.cn/1.0/redirect?url=http%3A%2F%2Fwww.leyou.com.cn%2Fproduct%2Fsingle%2FP810000200&item_id=P810000200&api_key=API_KEY&req_id=9950d641-4b4e-412e-b59d-35abbd06425f",
                "item_id": "P810000200"
            }, {
                "item_name": "\u8d1d\u4eb2--\u5a74\u513f\u624b\u53e3\u64e6\u62ed\u6e7f\u5dfe70\u72473\u5305\u4fc3\u9500\u88c5(\u65e0\u9152\u7cbe)",
                "price": "33.00",
                "image_link": "http://image.leyou.com.cn/images_db/09/1a/091a2951d7cf8868aae6ae1466472ce7.JPG",
                "market_price": "49.00",
                "score": 0.99559336956699995,
                "item_id": "603PL14500",
                "item_link": "http://api.leyou.com.cn/1.0/redirect?url=http%3A%2F%2Fwww.leyou.com.cn%2Fproduct%2Fsingle%2F603PL14500&item_id=603PL14500&api_key=API_KEY&req_id=9950d641-4b4e-412e-b59d-35abbd06425f"
            }, {
                "item_name": "\u4f0a\u745e\u4e1d--\u8d85\u6e29\u548c\u578b\u9ad8\u7aef\u624b\u53e3\u6e7f\u5dfe(80\u7247)",
                "price": "26.00",
                "image_link": "http://image.leyou.com.cn/images_db/03/86/0386b98f397d44562fa8b1624e036a07.JPG",
                "market_price": "32.00",
                "score": 0.995351572001,
                "item_id": "U290001300",
                "item_link": "http://api.leyou.com.cn/1.0/redirect?url=http%3A%2F%2Fwww.leyou.com.cn%2Fproduct%2Fsingle%2FU290001300&item_id=U290001300&api_key=API_KEY&req_id=9950d641-4b4e-412e-b59d-35abbd06425f"
            }, {
                "item_name": "\u8d1d\u4eb2--\u5a74\u513f\u67d4\u6e7f\u5dfe80\u72473\u8fde\u5305",
                "price": "33.00",
                "market_price": "49.00",
                "image_link": "http://image.leyou.com.cn/images_db/47/31/4731dfabd3ff767883b99e2d3543b283.JPG",
                "score": 0.99526662895200002,
                "item_link": "http://api.leyou.com.cn/1.0/redirect?url=http%3A%2F%2Fwww.leyou.com.cn%2Fproduct%2Fsingle%2F603PL13500&item_id=603PL13500&api_key=API_KEY&req_id=9950d641-4b4e-412e-b59d-35abbd06425f",
                "item_id": "603PL13500"
            }, {
                "item_name": "\u4f0a\u745e\u4e1d--\u690d\u7269\u914d\u65b9\u5a74\u513fPP\u4e13\u7528\u6e7f\u5dfe25\u7247\u00d74\u5305/\u888b",
                "price": "29.00",
                "market_price": "39.80",
                "image_link": "http://image.leyou.com.cn/images_db/eb/5e/eb5edd5770d1f97016e232a526096c14.JPG",
                "score": 0.99460654286299999,
                "item_link": "http://api.leyou.com.cn/1.0/redirect?url=http%3A%2F%2Fwww.leyou.com.cn%2Fproduct%2Fsingle%2FU290002101&item_id=U290002101&api_key=API_KEY&req_id=9950d641-4b4e-412e-b59d-35abbd06425f",
                "item_id": "U290002101"
            }],
            "code": 0,
            "req_id": "9950d641-4b4e-412e-b59d-35abbd06425f"
        },
        // CODES OMITTED
        ],
            "code": 0,
            "req_id": "86f15054-4f6c-48d2-82a8-65d112ca46b2"
        }
    }
})

The response is in plain json format. So it can be easily parsed and consumed.

(TODO)Dashboard

(TODO)Admin

(TODO)Batch

(TODO)Hadoop

Clone this wiki locally