Vision for this project

[grothesque]

Perhaps it is a good time to discuss the scope and vision of this project. So far I am only aware of this comment in which the author states that mdarray is merely a hobby experiment.

My impression is that already now this library has matured into a genuinely useful piece of infrastructure for numerical Rust. It combines the strengths of ndarray and nalgebra without giving up much (or anything?) in terms of performance or usability! While I can see many useful ways in which it could be extended (mostly by add-on crates I hope), I think that the core library is coherent, complete in a minimalist way, and elegant. Am I missing something important?

Although this is a small project and the bus factor is 1, I think that the unique compatibility guarantees of the Rust ecosystem make it already viable as a dependency for real-world software (In my case that’s research libraries and codes): even if the author was to stop development, and no one would take over, the compatibility guarantees of Rust and cargo would keep it usable and useful as a dependency for the foreseeable future - that is unless a critical security problem were to be found. Does this seem like a fair assessment, or am I missing some possible problems here?

Here are some specific points that might be of interest:

• What is the vision for the core mdarray crate? My hope is that it can be left without any dependencies except to fundamental infrastructure crates like serde. In this way its character as infrastructure would be preserved best. In particular, I think that this crate should not depend on any particular way of performing matrix multiplication or other non-trivial array operations. IMHO such functionality should be added by add-on crates, in order to avoid lock-in into a particular way of doing things.

• For most production applications, some form of BLAS/LAPACK will be needed. (For the application that I am interested in we need ways to perform matrix products, as well as SVD- and LU-decompositions.) Bindings to various forms of BLAS and LAPACK (and even native implementations) exist as Rust crates, so I guess that an mdarray-lapack crate (or several of them) could be created quite easily.

• I am also interested (to the point of being willing to work on it) in a sister library for sparse arrays. Not a library for sparse 2d matrices, but sparse multidimensional arrays, like https://sparse.pydata.org/ in the Python world. That Python library has the nice property of playing very well with NumPy. In many cases its sparse arrays can be substituted for dense NumPy arrays, and both can be mixed arbitrarily (sparse-dense products are efficient for example). I hope that a similar trick can be pulled off in Rust.

• Given that GPU computations are becoming more and more mainstream, what about GPUs in the context of mdarray? In the comment I linked at the very top, the author says that “the design in mdarray works well with arrays that are directly addressable on CPU”. How about an approach similar to CuPy, which is a NumPy-compatible library for arrays living in GPU address space. CuPy seems useful for large arrays: a similar approach in Rust ought to work much better (due to lower overhead and better parallelization). Any thoughts on how something like this could be best integrated with mdarray?

• Likewise, any thoughts on compatibility/conversion with other Rust library crates? ndarray and nalgebra in particular are used widely. Would conversion routines be useful? In which form?

• Is the author keen to advertise the existence of this library more widely to the community? Is it still too early?