Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use xinput to not depend on access to /dev/console (root only). #36

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

# ulogme


Expand All @@ -16,15 +15,15 @@ The project currently **only works on Ubuntu and OSX**, and uses new fancy **Pro

## Demo

See a blog post (along with multiple screenshots) describing the project [here.](http://karpathy.github.io/2014/08/03/quantifying-productivity/)
See a blog post (along with multiple screenshots) describing the project [here](http://karpathy.github.io/2014/08/03/quantifying-productivity/).

## Getting Started

**To start recording**

1. Clone the repository to some folder: `$ git clone https://github.com/karpathy/ulogme.git`
2. If you're on Ubuntu, make sure you have the dependencies: `$ sudo apt-get install xdotool wmctrl`. On Fedora, you may also need `sudo yum install gnome-screensaver`.
3. `cd` inside and run `$ ./ulogme.sh` (note: this will ask you for sudo authentication which is required for `showkey` command). This will launch two scripts. One records the frequency of keystrokes and the other records active window titles. Both write their logs into log files in the `logs/` directory. Every log file is very simply just the unix time stamp followed by data, one per line.
2. If you're on Ubuntu, make sure you have the dependencies: `$ sudo apt-get install xdotool xinput wmctrl`. On Fedora, you may also need `sudo yum install gnome-screensaver`.
3. `cd` inside and run `$ ./ulogme.sh` This will launch two scripts. One records the frequency of keystrokes and the other records active window titles. Both write their logs into log files in the `logs/` directory. Every log file is very simply just the unix time stamp followed by data, one per line.
4. For **OSX** only: there might be an additional step where you have to go to System Preferences > Security & Privacy > Accessibility, and make sure that Terminal (or iTerm2, or whatever you use to launch ulogme) is checked. If it wasn't checked previously and you just checked it, you may need to restart ulogme. If you don't do this step, you might find that window logging works but keypress logging doesn't.

**The user interface**
Expand All @@ -44,21 +43,21 @@ The user interface can switch between a single day view and an overview view by

#### Overview page

- You can click the window titles to toggle them on and off from the visualization
- You can click the window titles to toggle them on and off from the visualization
- Clicking on the vertical bars takes you to the full statistics for that day.

## Known issues
- One Ubuntu user reported broken view with no data. On further inspection we found that the logs were corrupt. One of the lines in a file in `/logs` was, instead of looking as `{timestamp} {data}` looked as `@@@@@@@{timestamp} {data}`, in other words an odd character was appended to the timestamp somehow. We manually erased these characters from the log file to fix the issue.
- Legacy code note: if you used ulogme from before 28 July, you will have to run `$ python legacy_split_events.py` to convert your events files, once.
- You may see *"address already in use"* if you try to run `python ulogme_serve.py`. Sometimes the system can get confused and takes a while to update what ports are being used. Use the optional argument to specify a different port, for example `python ulogme_serve.py 8124` and then go to `http://localhost:8124` instead, for example.
- Overview page is blank. Are you sure your browser supports ECMAScript 6? Chrome should be fine, Firefox might not be, yet.
- Overview page is blank. Are you sure your browser supports ECMAScript 6? Chrome should be fine, Firefox might not be, yet.

## Contributing

The Ubuntu and OSX code base are a little separate on the data collection side. However, they each just record very simple log files in `/logs`. Once the log files are written, `export_events.py` takes the log files, does some simple processing and writes the results into `.json` files in `/render`. The Javascript/HTML/CSS UI codebase is all common and all lives in `/render`.

### Ubuntu
ulogme has three main parts:
ulogme has three main parts:

1. Recording scripts `keyfreq.sh` and `logactivewin.sh`. You probably won't touch these.
2. Webserver: `ulogme_serve.py` which wraps Python's `SimpleHTTPServer` and does some basic communication with the UI. For example, the UI can ask the server to write a note to a log file, or for a refresh.
Expand Down
40 changes: 25 additions & 15 deletions keyfreq.sh
Original file line number Diff line number Diff line change
@@ -1,32 +1,42 @@
#!/bin/bash


# logs the key press frequency over 9 second window. Logs are written
# logs the key press frequency over 9 second window. Logs are written
# in logs/keyfreqX.txt every 9 seconds, where X is unix timestamp of 7am of the
# recording day.

LANG=en_US.utf8

helperfile="logs/keyfreqraw.txt" # temporary helper file
helperfile='/dev/shm/keyfreqraw.txt'
Copy link

@dcousens dcousens May 31, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not /tmp?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/dev/shm is almost always a tmpfs which is important for:

Never store highly sensitive information like keystrokes (which potentially can contain passwords) on persistent storage.

Ref: https://superuser.com/questions/45342/when-should-i-use-dev-shm-and-when-should-i-use-tmp

## FIXME: Still potential security risk even when not logged to persistent storage.
## Maybe rewrite script in Python and count STDOUT directly and send SIGINT
## after 9 seconds.

mkdir -p logs

trap 'kill $(jobs -p)' EXIT

while true
do
showkey > $helperfile &
PID=$!

# work in windows of 9 seconds
xinput test 11 > $helperfile &
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK when you have 2 keyboards (==laptop w. external one), you capture at most one with this.

For security purposes, I would rather pipe it to | tr -d '[0-9]' or so instead of /dev/shm

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For security purposes, I would rather pipe it to | tr -d '[0-9]' or so instead of /dev/shm

Prefect! Thanks very much 👍 Updated my PR. I think /dev/shm should be fine or is there something against using it?

AFAIK when you have 2 keyboards (==laptop w. external one), you capture at most one with this.

You are right. Thats when I saw your PR and intended it to be merged first so that I could rebase on it and steal that peace from you. I am to lazy to check that out right now but that is a todo.


## In case you can not get `xinput` to work. Note that you will need to run `showkey` as root.
# showkey > $helperfile &

# Work in windows of 9 seconds
sleep 9
kill $PID

# count number of key release events
num=$(cat $helperfile | grep release | wc -l)

# append unix time stamp and the number into file

# shellcheck disable=SC2046
kill $(jobs -rp)
# shellcheck disable=SC2046
wait $(jobs -rp) 2>/dev/null

# Count number of key release events
num="$(grep --count release "$helperfile")"

# Append unix time stamp and the number into file
logfile="logs/keyfreq_$(python rewind7am.py).txt"
echo "$(date +%s) $num" >> $logfile
echo "$(date +%s) $num" >> "$logfile"
echo "logged key frequency: $(date) $num release events detected into $logfile"

done

done
5 changes: 3 additions & 2 deletions ulogme.sh
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
#!/bin/bash

trap 'kill $(jobs -p)' EXIT

if [ "$(uname)" == "Darwin" ]; then
# This is a Mac
./osx/run_ulogme_osx.sh
else
# Assume Linux
sudo echo -n ""
sudo ./keyfreq.sh &
./keyfreq.sh &
./logactivewin.sh
fi