homework/Day2_homework_bigdata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TechX2022 - 计算社科 - HW2\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第一题 按chunk读取并分析数据\n",
    "\n",
    "- 下载ows-raw.txt (链接见作业页面)\n",
    "- 按照https://chengjun.github.io/mybook/06-data-cleaning-preprocessing.html 中介绍的按照chunk读取数据的方式读取数据\n",
    "- 以使用csv包为例：\n",
    "- 提示：from collections import defaultdict\n",
    "    - 提示：`day_user_dict = defaultdict(list)`\n",
    "- 对链接中代码的lines写for循环进行遍历，提取每一行的day和user\n",
    "    - `day_user_dict[day].append(user)`\n",
    "- 注意对for循环里的每一行写try except语句规避错误\n",
    "- 最后：\n",
    "\n",
    "```\n",
    "for i in day_user_dict:\n",
    "    print(i,len(day_user_dict[i]),len(set(day_user_dict[i])))\n",
    "```\n",
    "\n",
    "\n",
    "注意：请命名一个为 file_path 的字符串，其内容为 ows-raw.csv 的路径。在之后导入文件的时候直接使用这个变量，方便我们批改。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path = '/Users/qitianhu/Desktop/CSS-HW/Day1/ows-raw.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-06T09:52:57.119723Z",
     "start_time": "2020-08-06T09:52:04.956593Z"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-06T09:51:56.940736Z",
     "start_time": "2020-08-06T09:51:56.932876Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6602141"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. 打印出每天分别有几条tweets，几个 unique users. 提示：可以使用 csv package 的 csv.reader。需要对每一行字符串需要进行一些必要的替换和处理。同时，为了缓解内存压力，需要分批读取与处理（比如每次读取10万行），然后在存储并更新结果之后读取下一批。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-06T10:19:25.920892Z",
     "start_time": "2020-08-06T10:18:42.060042Z"
    },
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 100000\n",
      "1 200000\n",
      "2 300000\n",
      "2 400000\n",
      "2 500000\n",
      "3 600000\n",
      "3 700000\n",
      "4 800000\n",
      "4 900000\n",
      "4 1000000\n",
      "5 1100000\n",
      "5 1200000\n",
      "5 1300000\n",
      "6 1400000\n",
      "6 1500000\n",
      "7 1600000\n",
      "7 1700000\n",
      "7 1800000\n",
      "8 1900000\n",
      "8 2000000\n",
      "9 2100000\n",
      "9 2200000\n",
      "9 2300000\n",
      "10 2400000\n",
      "10 2500000\n",
      "10 2600000\n",
      "11 2700000\n",
      "11 2800000\n",
      "12 2900000\n",
      "12 3000000\n",
      "12 3100000\n",
      "13 3200000\n",
      "13 3300000\n",
      "13 3400000\n",
      "14 3500000\n",
      "14 3600000\n",
      "15 3700000\n",
      "15 3800000\n",
      "15 3900000\n",
      "16 4000000\n",
      "16 4100000\n",
      "17 4200000\n",
      "17 4300000\n",
      "17 4400000\n",
      "18 4500000\n",
      "18 4600000\n",
      "18 4700000\n",
      "19 4800000\n",
      "19 4900000\n",
      "20 5000000\n",
      "20 5100000\n",
      "20 5200000\n",
      "21 5300000\n",
      "21 5400000\n",
      "22 5500000\n",
      "22 5600000\n",
      "22 5700000\n",
      "23 5800000\n",
      "23 5900000\n",
      "23 6000000\n",
      "24 6100000\n",
      "24 6200000\n",
      "25 6300000\n",
      "25 6400000\n",
      "25 6500000\n",
      "26 6600000\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-08-06T10:19:48.477736Z",
     "start_time": "2020-08-06T10:19:47.987167Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "day, #tweets, #users\n",
      "2011-10-06 49638 18487\n",
      "2011-10-07 65238 23460\n",
      "2011-10-08 65949 23243\n",
      "2011-10-09 65097 22576\n",
      "2011-10-10 78619 27499\n",
      "2011-10-11 67596 22955\n",
      "2011-10-12 67753 25535\n",
      "2011-10-13 98954 39200\n",
      "2011-10-14 148062 47666\n",
      "2011-10-15 161802 48951\n",
      "2011-10-16 165381 51602\n",
      "2011-10-17 113628 41525\n",
      "2011-10-18 103337 37394\n",
      "2011-10-19 88874 34990\n",
      "2011-10-20 73159 29254\n",
      "2011-10-21 75256 27029\n",
      "2011-10-22 70368 24213\n",
      "2011-10-23 64808 22378\n",
      "2011-10-24 64571 24268\n",
      "2011-10-25 74641 27084\n",
      "2011-10-26 141729 46424\n",
      "2011-10-27 137172 43368\n",
      "2011-10-28 87216 31215\n",
      "2011-10-29 75741 25319\n",
      "2011-10-30 77813 24979\n",
      "2011-10-31 64256 25911\n",
      "2011-11-01 62286 23962\n",
      "2011-11-02 74554 26559\n",
      "2011-11-03 92698 32405\n",
      "2011-11-04 63529 24288\n",
      "2011-11-05 65019 22710\n",
      "2011-11-06 55511 21374\n",
      "2011-11-07 52820 20244\n",
      "2011-11-08 47453 18503\n",
      "2011-11-09 46503 17997\n",
      "2011-11-10 52664 19455\n",
      "2011-11-11 48997 18746\n",
      "2011-11-12 38189 13954\n",
      "2011-11-13 45470 14305\n",
      "2011-11-14 59815 20412\n",
      "2011-11-15 409075 98934\n",
      "2011-11-16 126520 45483\n",
      "2011-11-17 279349 73465\n",
      "2011-11-18 154395 52665\n",
      "2011-11-19 107747 37741\n",
      "2011-11-20 99365 37372\n",
      "2011-11-21 87818 34378\n",
      "2011-11-22 81802 32225\n",
      "2011-11-23 61471 25414\n",
      "2011-11-24 40522 16744\n",
      "2011-11-25 40930 16143\n",
      "2011-11-26 44523 16850\n",
      "2011-11-27 42614 16245\n",
      "2011-11-28 53505 19715\n",
      "2011-11-29 40152 15850\n",
      "2011-11-30 59289 20585\n",
      "2011-12-01 48062 17839\n",
      "2011-12-02 43135 16297\n",
      "2011-12-03 35760 13026\n",
      "2011-12-04 42326 13280\n",
      "2011-12-05 42429 14968\n",
      "2011-12-06 46917 16180\n",
      "2011-12-07 45440 16491\n",
      "2011-12-08 40733 15999\n",
      "2011-12-09 43030 15187\n",
      "2011-12-10 33987 11427\n",
      "2011-12-11 26604 9220\n",
      "2011-12-12 50448 15111\n",
      "2011-12-13 39982 13478\n",
      "2011-12-14 39399 13290\n",
      "2011-12-15 36426 13318\n",
      "2011-12-16 28107 10043\n",
      "2011-12-17 37983 10265\n",
      "2011-12-18 29022 9576\n",
      "2011-12-19 20893 8787\n",
      "2011-12-20 24045 8500\n",
      "2011-12-21 20091 7487\n",
      "2011-12-22 21972 7423\n",
      "2011-12-23 19031 6619\n",
      "2011-12-24 14196 5289\n",
      "2011-12-25 10538 3943\n",
      "2011-12-26 12515 4461\n",
      "2011-12-27 16772 6893\n",
      "2011-12-28 19117 6583\n",
      "2011-12-29 17100 6094\n",
      "2011-12-30 17367 5975\n",
      "2011-12-31 14692 5070\n",
      "2012-01-01 33192 9486\n",
      "2012-01-02 19529 7009\n",
      "2012-01-03 22790 7299\n",
      "2012-01-04 16827 6391\n",
      "2012-01-05 11227 4493\n",
      "2012-01-06 10770 4324\n",
      "2012-01-07 14432 5183\n",
      "2012-01-08 15209 5400\n",
      "2012-01-09 16589 5795\n",
      "2012-01-10 16829 5975\n",
      "2012-01-11 22665 7722\n",
      "2012-01-12 17705 7843\n",
      "2012-01-13 15683 6727\n",
      "2012-01-14 15046 5416\n",
      "2012-01-15 16264 5931\n",
      "2012-01-16 15844 5971\n",
      "2012-01-17 20692 7331\n",
      "2012-01-18 18162 7412\n",
      "2012-01-19 14533 5948\n",
      "2012-01-20 19840 8133\n",
      "2012-01-21 15459 5514\n",
      "2012-01-22 11948 4502\n",
      "2012-01-23 14896 5989\n",
      "2012-01-24 14290 5360\n",
      "2012-01-25 18812 8440\n",
      "2012-01-26 15159 5938\n",
      "2012-01-27 11992 4758\n",
      "2012-01-28 11530 4243\n",
      "2012-01-29 27219 8856\n",
      "2012-01-30 27198 9038\n",
      "2012-01-31 18682 7120\n",
      "2012-02-01 16578 5958\n",
      "2012-02-02 14995 5452\n",
      "2012-02-03 14371 5087\n",
      "2012-02-04 16591 5443\n",
      "2012-02-05 13656 4603\n",
      "2012-02-06 14446 5201\n",
      "2012-02-07 14940 5244\n",
      "2012-02-08 13451 5099\n",
      "2012-02-09 13944 5156\n",
      "2012-02-10 14083 4843\n",
      "2012-02-11 12920 4616\n",
      "2012-02-12 11178 4131\n",
      "2012-02-13 13600 4799\n",
      "2012-02-14 13580 4669\n",
      "2012-02-15 13248 4895\n",
      "2012-02-16 12837 4428\n",
      "2012-02-17 12468 4299\n",
      "2012-02-18 4859 2012\n"
     ]
    }
   ],
   "source": [
    "# 打印答案"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 题目2 基于占领华尔街推特数据预测被警察逮捕人数 (选做，非必选)\n",
    "\n",
    "- 首先阅读：Mooijman, M., Hoover, J., Lin, Y. et al. Moralization in social networks and the emergence of violence during protests. Nat Hum Behav 2, 389–396 (2018). https://doi.org/10.1038/s41562-018-0353-0\n",
    "- 下载并读取OWS被捕人数数据 https://www.stpete4peace.org/occupyarrests 参考：https://chengjun.github.io/mybook/06-data-cleaning-preprocessing.html 需要了解的内容：大数据的读取、清洗、文本挖掘的基础、机器学习的基本使用、基于时间序列的机器学习的逻辑 \n",
    "- 提取ows tweets数据中的arrest、cops、police、nypd等词每天出现的次数\n",
    "- 分析这些词语在每天的词频与每天实际被捕人数之间的关系"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}