Skip to content

Commit

Permalink
Added guidelines for WorldBrain's Webmemex (#10)
Browse files Browse the repository at this point in the history
* Added writeup for WorldBrain's Webmemex

* Added WorldBrain's Webmemex to the list

* Grammatical and spelling fixes
  • Loading branch information
bohrium272 authored and nikhita committed Aug 30, 2017
1 parent 11e5457 commit 2a9ea61
Show file tree
Hide file tree
Showing 2 changed files with 176 additions and 0 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ Please follow the [contributing guide](CONTRIBUTING.md) to add your own contribu
| [Ceph](http://ceph.com/) | [Multilanguage RGW Testing](writeups/Ceph_Multilaguage-RGW-Testing_JoannahNanjekye.md) | [Joannah Nanjekye](https://github.com/nanjekyejoannah) |
| [Cloud Native Computing Foundation](https://www.cncf.io/) | [Kubernetes](writeups/CNCF_Kubernetes_NikhitaRaghunath.md) | [Nikhita Raghunath](https://github.com/nikhita) |
| [phpMyAdmin](https://www.phpmyadmin.net/) | [phpMyAdmin](writeups/phpmyadmin_phpmyadmin_manishbisht.md) | [Manish Bisht](https://github.com/manishbisht) |
| [WorldBrain](https://worldbrain.io/) | [WebMemex](writeups/worldbrain_webmemex_arpitgogia.md) | [Arpit Gogia](https://github.com/arpitgogia) |
175 changes: 175 additions & 0 deletions writeups/worldbrain_webmemex_arpitgogia.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
# WorldBrain's WebMemex

A browser extension that lets you create, search and share your personal web of knowledge.

- **Full-Text Search** pages that you visited and bookmarked (and later) also all the apps you use to organise your knowledge.
- **Manually and automatically create links** between your content to organise & search your web by *your associations*
- **Share** content recommendations, associations and qualifying metadata with your network and followers


## Project status

Sorry to disappoint, but we do not have the promised solution complete yet. :unamused:

In its current released version, you can full-text search the browsing history & bookmarks.
You can download it here: [worldbrain.io/download](http://worldbrain.io/download)

At the moment we are in the process of refactoring/porting the features of the [WorldBrain (Re)search-Engine](https://github.com/WorldBrain/Research-Engine/) into the *WebMemex*, a project started by [@treora](https://github.com/Treora)

This project is in full development. You are most welcome to contribute! See
[Hacking](#hacking) below about how to build and run it from source and make it learn new tricks.


## Approach

As it stands now, it is a WebExtension (thus should work on most modern
browsers), bundled by [browserify](http://browserify.org) with some
[babel](https://babeljs.io) ES6–7→ES5 compilation, that logs and stores visited
pages in [PouchDB](https://pouchdb.com), and provides a viewer for this data
based on
[React](https://facebook.github.io/react/)+[Redux](http://redux.js.org/). See
[Code Anatomy](#code-anatomy) below for a full tour du code.

The project strategy is to combine and integrate features from other projects,
and to factor out developed functionality into separate modules wherever it
seems sensible. This extension could then be regarded as a bunch of different
(but related) features a browser ought to have, bundled together for quick
installation.

See [our initial blog post](https://blog.webmemex.org/2017/01/05/roadmap/)
for the feature roadmap of the next few months.


## Contribute

We are happy about any kind of feedback, bug fixes, new feature ideas and tips!
Give a shout. :loudspeaker:

- [Drop by on Slack](http://join-worldbrain.herokuapp.com)
- Write us an email: [email protected]
- or leave a comment under issues you want to contribute to


## Hacking

Like playing with ES6, WebExtension browser APIs, React, Redux, PouchDB? Come
play along! :tada:

### Build and run it

1. Clone this repo.
2. Get [Node/NPM](https://nodejs.org) and [yarn](https://yarnpkg.com)
(`npm install -g yarn`).
3. Run `yarn` to install dependencies.
4. Compile the source files using `yarn watch` for incremental builds or `yarn build` for standard dev builds.
5. Load it in Firefox or Chromium/Chrome:
* In Firefox (≥49): run `yarn run firefox` (or run [web-ext] directly for more control).
Alternatively, go to `about:debugging`, choose 'Load Temporary Add-on', and pick
`extension/manifest.json` from this repo.
* In Chromium/Chrome: go to Tools→Extensions (`chrome://extensions`), enable 'Developer mode',
click 'Load unpacked extension...', and pick the `extension/` folder from this repo.

### Automatic recompilation

If the steps above worked, running `yarn run watch` will trigger a quick
recompilation every time a source file has been modified.

If you are testing in Firefox through `yarn run firefox`, the extension should also reload
automatically. Otherwise, manually press the reload button in the extension list.

- If your edits affected only the overview interface, just refresh/reopen it.
- However, if you changed the background script, you have will to reload the
extension: find it back in the list of temporary add-ons/extensions and click
Reload.
- If you changed the 'content_script', it seems browser-dependent whether newly
opened pages will get the new version. Better reload the extension to be sure.


## Code Anatomy

### WebExtension parts

To comply with the [anatomy of a WebExtension](https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Anatomy_of_a_WebExtension),
it consists of three main parts (found in [`extension/`](extension/) after
compilation):

- `background.js` always runs, in an 'empty invisible tab', listening for
messages and events.
- `content_script.js` is loaded into every web page that is visited. It is
invisible from that web page's own scripts, and can talk to the background
script.
- `overview/overview.html`, with the resources in that folder, provides the main
user interface.

The parts communicate in two ways:
- Messaging through `browser.sendMessage`, usually done implicitly by using a
remote procedure call ([`util/webextensionRPC.js`](src/util/webextensionRPC.js)).
- Through the in-browser PouchDB database, they get to see the same data, and
can react to changes made by other parts.

Besides these parts,
[`browser-polyfill.js`](https://github.com/mozilla/webextension-polyfill/)
provides the promise-based `browser` API, that simply wraps Chromium/Chrome's
callback-based `chrome` API, in order to make the same code run in different
browsers (and to structure the callback mess).

### Source organisation

To keep things modular, the source code in [`src/`](src/) is not split in
exactly those the three parts of the extension, but is rather grouped by
functionality. Some folders may end up being factored out into separate repos
later on, or at some point perhaps even into separate but interacting browser
extensions.

#### [`src/activity-logger/`](src/activity-logger/): activity logger

This logs every page visit in PouchDB. Soon it should also watch for user
interactions, for example to remember which parts of a page you have read.

Currently, for every visit, a new page object is created in the database, to
represent the visited page itself. This object should soon be deduplicated when
the same page is visited multiple times. After creating a new page object,
the next module is triggered to start analysing the page.

#### [`src/page-analysis`](src/page-analysis/): page analysis

This extracts and stores information about the page in a given tab, such as:
- The plain text of the page, mainly for the full-text
search index.
- Metadata, such as its author, publication date, etcetera.
- A screenshot for visual recognition.

#### [`src/overview/`](src/overview/): overview

The `overview` is the user interface that opens in a tab of its own. It is built
with [React](https://facebook.github.io/react/) and [Redux](http://redux.js.org/),
which create a somewhat complex but nicely organised application structure.

See [`src/overview/Readme.md`](src/overview/Readme.md) for more details.

#### [`src/search`](src/search/): document search

Functions for finding relevant knowledge in the user's memory. Currently
provides a simple full-text keyword search through visited pages using
[pouchdb-quick-search](https://github.com/nolanlawson/pouchdb-quick-search).
This can be improved in many ways, because we are searching through a person's
memory, not just some arbitrary document collection. For example, we can use
created associations and browsing paths to better understand what one is looking
for.

#### [`src/dev/`](src/dev/): development tools

Tools to help during development. They are not used in production builds.

#### [`src/util/`](src/util): utilities

Contains small generic things, stuff that is not project-specific. Things that
could perhaps be packaged and published as an NPM module some day.

#### `...`: other stuff

The build process is based on `yarn`, that runs some `npm` commands specified in
`package.json`, which in turn start the corresponding tasks in
`gulpfile.babel.js` (transpiled by settings in `.babelrc`).

So much for the code tour. :zzz: Any questions? :point_up:

0 comments on commit 2a9ea61

Please sign in to comment.