Dashboard Compiler & Publisher

[domoritz]

Given a Mosaic YAML/JSON specification (and possibly additional data source information), analyze the spec and produce a standalone dashboard web application. The compiler should analyze the data sources and convert them for rapid loading within a WASM-based DuckDB deployment. The compiler should also raise warnings or exit with an error if the dataset is too big to export to a standalone web-based deployment. The compilation process might also consider data cube indexing, for example by precomputing cube tables and adding them to DuckDB upon dashboard initialization. The compiler might also suggest dashboard modifications when helpful, for example by rewriting a selection to use single resolution if cube computation will be too costly at runtime. A dashboard publishing tool like Observable Framework might be one way to then deploy the results.

[jheer]

I've started some initial work on this, so if others are interested in pitching in please get in touch. Or, just "thumbs up" vote on the entry above if this is something you'd like to see supported.

[nicosuave]

I'm interested in helping

[alexkreidler]

I built a tool to edit and live-preview Mosaic specs in your browser: https://mosaic-playground.netlify.app/. It doesn't generate a static site, but I've been using it to create Mosaic JSON specs that I then put in an Observable Framework markdown file.

I'd love some feedback and would also be open to contribute it to Mosaic: https://github.com/alexkreidler/mosaic-playground

It would be awesome to have a way to pre-bake data for initial views (#506), or an option to render Mosaic to a non-interactive static SVG that can be used in a static site generator like in this Observable Framework example.

Thank you for your awesome work! I'm excited to see what everyone is cooking up for this issue!

[jheer]

Wow, this looks great @alexkreidler! Thanks for developing and sharing this.

I'd definitely be interested in folding some variant of this into the Mosaic project proper (e.g., as a new package within the monorepo). In terms of constructive feedback, it would be nice to include support for YAML and for JSON schema auto-complete. I also notice a rather aggressive stream of red error notifications in the midst of normal typing in the editor.

FWIW, I also plan to further develop both pre-baked data and server-side SVG rendering, but need to clear a number of other obligations off my plate before diving into those. In any case, thanks again for sharing!

[alexkreidler]

@jheer I've improved the error notifications and added YAML support and JSON autocomplete to mosaic-playground (there are some issues blocking autocomplete for YAML).

On the static rendering of Mosaic, I wrote this Typescript example that uses JSDom and the Node duckdb library to render the plot in NodeJS:

import * as vg from "@uwdata/vgplot";
import { DuckDB } from "@uwdata/mosaic-duckdb";
import { decodeIPC } from "@uwdata/mosaic-core";
import { astToDOM, parseSpec } from "@uwdata/mosaic-spec";
import 'global-jsdom/register'
import spec from "./line.json";

// Otherwise, `new Event()` which is used by some Observable Plot code will use the Node.js Event class
// which causes errors because JSDom needs its own browser Event class.
global.Event = window.Event;

// From https://github.com/uwdata/mosaic/blob/main/packages/core/test/util/node-connector.js
export function nodeConnector(db = new DuckDB()) {
  return {
    /**
     * Query an in-process DuckDB instance.
     * @param {object} query
     * @param {string} [query.type] The query type: 'exec', 'arrow', or 'json'.
     * @param {string} query.sql A SQL query string.
     * @returns the query result
     */
    query: async (query) => {
      const { type, sql } = query;
      switch (type) {
        case "exec":
          return db.exec(sql);
        case "arrow":
          return decodeIPC(await db.arrowBuffer(sql));
        default:
          return db.query(sql);
      }
    },
  };
}

async function specToSVG(spec: any): Promise<string> {
  const coord = vg.coordinator()
  coord.databaseConnector(nodeConnector());
  // coord.manager.logQueries(true);
  const ast = parseSpec(spec);
  
  const { element } = await astToDOM(ast);
  // value is a reference to the @uwdata/mosaic-plot Plot instance
  const sync = element.value.synch;
  // wait for all queries to complete and the plot to re-render
  await sync.promise;
  return element.outerHTML;
}

specToSVG(spec).then(console.log).catch(console.error);

It works well except the tooltip doesn't work.

I can think of one way to pre-bake data following this approach: modify the nodeConnector to store <query, result> pairs in some format (maybe a JSON file mapping queries to IDs/file names of a bunch of arrow files, which can then be used by the connector or DuckDB-wasm) DuckDB-wasm itself (plus extensions) has a large bundle size that is probably a big part of startup time, so maybe it would be possible for the WASM connector to be ready to serve queries from that static arrow file cache while WASM is loading for more filtering/interaction. IDK how this would interact with the current caching system.

[edublancas]

hey all, I've built a tool to generate mosaic specs from a UI: ploomber.github.io/mosaic-spec-generator

once the plot is generated, you can view the spec and sample code for creating Dash and Streamlit apps - pretty basic but would love to hear your feedback.

I think generating a standalone dashboard is a great idea, I'll see if I find some time to implement that.

P.S. there's a bug that causes the app not to show any errors if astToDOM fails, still trying to figure out why I'm unable to catch the exception, will ask for feedback soon.

[jheer]

At @edublancas, that's really cool, thanks for sharing! As for the bug, I wonder if there is an error being thrown somewhere in a Promise context and it's not getting caught? If so, definitely something we want to fix.

There is also the nice work @alexkreidler has done on a Mosaic editor. I wonder if it makes sense to "join forces" and plot out an "official" supported editor as part of the Mosaic project? At minimum, I could imagine a number of shared components or subroutines that could be valuable across such efforts.

[alexkreidler]

@edublancas Very nice! Is it open source?

@jheer I'm happy to join forces! I could open an issue to discuss the editor or send a pull request with the current mosaic-playground code. Then people could re-use components and logic (like exporting to SVG/PNG) from a package in the repo to build a dashboard tool or visual Mosaic editor.

[edublancas]

@alexkreidler it's not open-source (we might do it in the future, though), but I'm ok giving you access to the repo if you think this could help you

@jheer, yeah, I'd love to collaborate. I cannot commit much time (I did this as a small side project for my clients), but I'm happy to share my source code, collaborate on small patches/features, and give feedback.

[declann]

The Mosaic Playground by @alexkreidler is impressive 👏 , and a really great tool to study/manipulate different specs. Share functionality could also be cool - such as in the Vega Editor.