I avoided discussing tools in the book because tooling is ephemeral -- all software will eventually become legacy software. However, I did mention a few tools in my book when they were useful to illustrate a point. Several early readers told me that the book helped them discover several useful tools, and asked me to recommend more, so here is my attempt.
In this doc, I’ll start with long lists of tools other people have compiled. Then we’ll go over a short list of open-source tools that I think are pretty cool. I exclude some of the already well known tools in the data science community to keep the list short. Feel free to open an issue or submit a PR to recommend a new tool. Thank you!
We also have a channel (#tools-watch) on our MLOps Discord to discuss interesting tools.
I get it -- there are hundreds of MLOps tools out there. There have been multiple long lists of MLOps tools, each with hundreds so I won’t create another list here. However, if you want a long list of tools, here are some of the more popular resources.
Notes
- Many of these lists were created by vendor-sponsored communities. There’s nothing wrong with taking sponsorships -- communities have operational expenses too -- and being able to get sponsorships is impressive for a community. However, sometimes, there might be conflicts of interests between vendor-sponsorships and vendor-neutral reviews.
- Many of these lists divide tools into categories and evaluate them by features. However, the landscape is still evolving quickly, so categories are not yet well-defined. At the same time, best practices are still converging, so it can be hard for companies to delineate what features they need.
- MLOps Community’s Learn: currently, it consists of 3 categories (Feature Store, Monitoring, Metadata Storage & Management -- which includes Model Store) with one in progress (Deploy). You can add 2 tools to compare them side-by-side.
- TwiML Solutions Guide: Tools are grouped into categories with features supported and where they can be deployed.
- StackShare: You can browse different devtools, see what companies are using them, see jobs that mention them, check out alternatives, even read reviews. You can also see the tech stacks of popular tech companies. You can also compare tools side by side. In my opinion, StackShare is what many MLOps solution guides aspire to become, but StackShare is general for all tech stacks, not just MLOps.
- AI Infrastructure Landscape: a membership-based foundation that focuses on AI infrastructure (more general than LFAI & Data, which focuses on open-source). They do a lot of comprehensive research and analysis on MLOps tooling.
- Matt Turck’s annual Machine Learning, AI and Data (MAD) Landscape: with high-level analysis from an investor’s perspective.
- Leigh Marie Braswell’s Startup Opportunities in Machine Learning Infrastructure: analysis of the gaps in MLOps landscape from another investor’s perspective.
- Yours truly’s MLOps Landscape. It comes with a Google Sheets list of 284 tools (last updated in December 2020) and interactive visualization. I stopped maintaining the list because it takes a ton of time and it’s not like no one else is doing it.
I star cool open source tools whenever I come across them. You can check out the full list here.
Disclaimer: this is not supposed to be a comprehensive list.
Table formats like Pandas DataFrame are great for data manipulation. However, Pandas is slow, quirky, and doesn’t leverage GPUs. Naturally, there have been many projects aiming to fix these problems. Here are a few of them.
- modin: a drop-in replacement of pandas (using
import modin.pandas as pdinstead ofimport pandas as pd). Ponder is working on the enterprise version built on top of it. - dask and cuDF: both projects were at some point maintained by the RAPIDS AI team at NVIDIA, until a core developer of dask left NVIDIA to start Coiled, which provides the enterprise version built on top of dask.
- Polars: built with Rust on top of Apache Arrow, Polars promises to be fast!
H2O did a fun database-like ops benchmark (though I hope they’ll consider another color scheme for their graphs).
- Online analytical databases (e.g. if you want to join predictions with user feedback to monitor your model performance in real-time): ClickHouse, Druid.
- Stream processing: ksql, faust, materialize, Redpanda (on WASM)
- Visualization: D3, Superset, Facets, redash, visdom, plotly, Altair, pandas-profiling, lux (only for DataFrames), bokeh
- Data validation: Great Expectations, deepchecks, pandera
- Labeling: Snorkel, Label Studio, doccano
- Data versioning: DVC, Dolt, pachyderm
- Data hosting for unstructured data: Hub
- Metadata stores (for feature discovery): Lyft’s Amundsen and LinkedIn’s DataHub.
- Feature stores: FEAST (OSS). Several commercial feature stores are built on top of FEAST.
- Data pipelines: ploomber, hamilton, NVTabular
- The whole arrow and flight ecosystem is growing fast!
- Interpretability: SHAP, Lime, Interpret, lit (for NLP), captum (PyTorch), timeshap, AIX360
- Fairness: AIF360
- Experiment tracking: MLflow, aim. Most of the tools in this category aren’t open source since the hard part is to host and visualize artifacts.
- Model optimization: TVM, TensorRT, Triton, hummingbird, composer
- Distributed training: DeepSpeed (super cool), accelerate
- Federated learning: PySyft, FedML, FATE, TensorFlow Federated
- Evaluation: checklist (NLP), reclist (recommender systems)
- Online experiments (e.g. A/B testing): growthbook, Ax
Just me nerding out on those cool use cases -- don’t mind me.
- Neural recommender systems / CTR: DLRM, DeepCTR, tensorflow-DeepFM, Transformers4Rec
- Conversational AI: rasa, NeMo
- Similarity search: annoy, Faiss, Milvus
- Deep fakes: faceswap, deepface
- Time-lagged conversion modeling: convoys
- Churn prediction: WTTE-RNN
- Survival analysis: lifelines
- CLI tools: fzf (fuzzy search), lipgloss
- IDEs: just use VSCode -- its notebook is pretty good tooo.
- Dependency management: Poetry
- Config management: Hydra, gin-config
- Documentation: docusaurus
- Debugging in Kubernetes: k9s
- Virtual whiteboard: excalidraw
- CI/CD: earthly
- Monitoring: Sentry, Prometheus, vector, M3
- Dashboards: Grafana, Metabase
- General DevOps: Chaos Monkey, k6