2.1.1.3. Website Project Section: Vizualize Dat ecosystem projects & modules

[serapath]

#8

@todo

• crawl and scrape all repos from organisations and people in the dat-ecosystem
  • ask consortium to define a list of blessed dat-ecosystem stack modules
    • @output 📦 ./data/blessed.json
    • next write crawler/scraper to get the data #34
  • write crawler/scraper to get the data #34
  • Using previously collected projects and people information, categorize and organize the information as an infographic of the whole dat ecosystem.
    • @input 📦 ./<timestamp>/valuenetwork.json from write crawler/scraper to get the data #34
    • @input 📦 ./<timestamp>/packages.json from write crawler/scraper to get the data #34
    • @input 📦 ./<timestamp>/organisations.json from write crawler/scraper to get the data #34
    • @input 📦 ./<timestamp>/index.json from write crawler/scraper to get the data #34
    • @input 📦 ./index.json from write crawler/scraper to get the data #34
    • @output ❓ code that loads JSON ecosystem data and converts it to the formats required by graph visualization engines (these are not standardized but look like e.g. { nodes: [...], links: [...]})
      • first prototype: directly convert data into the graph visualizer necessitated format
      • stretch goal 1: using quadstore RDF engine to enable filtering
      • stretch goal 2: support application specific link/node data for visualizing nodes and edges differently
        • with the following application in mind: selecting and filtering dependency subtree to visualize trickles
      • stretch goal 3: streamed rendering for better UI performance
    • @output ❓ early prototype visualization implementation using 3d-force-graph or similar with default rendering settings
      • check in with @serapath @ninabreznik and others once this is done to check direction and evaluate priorities before implementing a final version
    • design a project card template per tier and website section template
    • annotate each repo in the template with maintainer projects/people
  • implement final code for generating info-viz
    • @input answers to open questions:
      • how does prototype visualization look and feel?
      • what's a satisfactory UI that's within scope?
      • what should we change?
      • what to prioritize next?
    • @output ❓ code for generating info-viz
  • Finally add the section to the website
    • @output ❓ design template component for card and website section - SVG output desirable. This would help us to add future projects and/or persons to the ecosystem map.
    • @output ❓ infographic asset of the whole dat-ecosystem with projects and important members. Ideally, projects and people should include logos and/or publicly available profile pictures. They should also contain important links (ie: projects links, websites, etc).

---

@info

Goal: help new or existing member easily find and explore relevant projects/orgs/modules in the ecosystem without having to manually update this info

The goal is to have a well-looking, clear, big view of the ecosystem, its projects and members.
Some example/similar projects:

• https://twitter.com/n2ckchong/status/1373533273398243328
• https://ecosystem.ipfs.io/
• The IPFS Ecosystem Diagram ipfs/awesome-ipfs#304

estimated duration:

5 days

estimated budget:

1600 usd

[serapath]

INSPIRATION

• https://fediverse.party/
• https://npmgraph.js.org/?q=hypercore
• https://github.com/npmgraph/npmgraph

[ninabreznik]

CONCEPT

• use dataset from scraper
• summary dashboard (x modules, y stars, range from 2013 to today, total number of collaborators...)
• filtering (by stars, by dependants, by contributors, by organization) + search

TODO

• get data = finish scraper (fix rate limits) OR scrape socket.dev instead (live or stored on github and rescraped in intervals)
• make table with filtering and summary (use data table framework)

[micahscopes]

I put a lot (>5 days) into this visualization and am satisfied with where it's at, so I'd like to consider the first push finished.

It was challenging to find existing graph visualization tools that were both aesthetically flexible and had good mobile performance, so I got sort of carried away and ended up making something from scratch.

Currently the simplest way to integrate this into the website would be to use an iframe. The visualization supports hash URLs (including the back button!) so it should be possible to interact with the graph just by setting a hash location from the parent page, for example by clicking on links in the parent document.

Since it'd probably be useful for lots of other projects too, I'm working on spinning the graph visualization engine into its own project called "Webscape Wanderer". Hopefully I can get some support to keep working on it! I'm also working on a blog post for the Dat Ecosystem blog, to share the project and the story of how it was built.

The visualization code is here, along with implementation details and a roadmap: https://github.com/micahscopes/dat-garden-visualization
A live version is here: https://micahscopes.github.io/dat-garden-visualization

Since I'm considering renaming and moving the micahscopes/dat-garden-visualization project, it could make sense to fork it to the dat-ecosystem org.